ConcurrentRebalancersTest.TwoConcurrentRebalancers/0: /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/mini-cluster/external_mini_cluster.cc:947: Failure
Expected equality of these values:
  0
  num_crashes
    Which is: 2
At least one process crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/rebalancer_tool-test.cc:1284: Failure
Expected: cluster_->AssertNoCrashes() doesn't generate new fatal failures in the current thread.
  Actual: it does.
I20240502 02:01:31.432195 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 5982
I20240502 02:01:31.439666  6109 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 02:01:31.488962  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 02:01:31.568082  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:01:31.576170  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 02:01:31.663688  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 02:01:31.825172 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 5982
W20240502 02:01:31.834472  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:01:31.883180  8204 connection.cc:535] server connection from 127.25.109.129:59983 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 02:01:31.883401  5905 connection.cc:535] server connection from 127.25.109.129:40579 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 02:01:31.883450  8204 connection.cc:535] client connection to 127.25.109.129:36227 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 02:01:31.883467  6402 connection.cc:535] client connection to 127.25.109.129:36227 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 02:01:31.883558  8204 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 02:01:31.884336  6401 connection.cc:535] server connection from 127.25.109.129:53907 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 02:01:31.884568 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 6119
W20240502 02:01:31.884824  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227): Couldn't send request to peer 218bc8c1f9ce46059898fc738179adf9. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:36227: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 02:01:31.885342  8204 consensus_peers.cc:485] T 89f7cf12d85442749e7d5b259ebca43f P 436822a397664c54b8d1b9eb3bc6cff4 -> Peer 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227): Couldn't send request to peer 218bc8c1f9ce46059898fc738179adf9. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:36227: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 02:01:31.890964  6243 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 02:01:31.922886  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227): Couldn't send request to peer 218bc8c1f9ce46059898fc738179adf9. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:36227: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 02:01:31.960361  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 02:01:32.279454 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 6119
W20240502 02:01:32.283303  6270 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227): Couldn't send request to peer 218bc8c1f9ce46059898fc738179adf9. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:36227: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 02:01:32.302795  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:01:32.334240  8204 connection.cc:535] server connection from 127.25.109.130:50235 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 02:01:32.334830 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 6253
I20240502 02:01:32.341012  6374 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 02:01:32.691327  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 02:01:32.698307  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 02:01:32.708259  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:01:32.734413 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 6253
W20240502 02:01:32.791685  7936 connection.cc:535] server connection from 127.25.109.131:40629 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 02:01:32.792347 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 6385
I20240502 02:01:32.796547 28953 raft_consensus.cc:482] T 5be45a68cff24a75a03b2fcd4b7851f9 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 218bc8c1f9ce46059898fc738179adf9)
I20240502 02:01:32.796752 28953 raft_consensus.cc:504] T 5be45a68cff24a75a03b2fcd4b7851f9 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 02:01:32.797780 28953 leader_election.cc:288] T 5be45a68cff24a75a03b2fcd4b7851f9 P 436822a397664c54b8d1b9eb3bc6cff4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227), a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197)
I20240502 02:01:32.798923  6509 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 02:01:32.799327  8204 leader_election.cc:334] T 5be45a68cff24a75a03b2fcd4b7851f9 P 436822a397664c54b8d1b9eb3bc6cff4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227): Network error: Client connection negotiation failed: client connection to 127.25.109.129:36227: connect: Connection refused (error 111)
W20240502 02:01:32.799542  8205 leader_election.cc:334] T 5be45a68cff24a75a03b2fcd4b7851f9 P 436822a397664c54b8d1b9eb3bc6cff4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197): Network error: Client connection negotiation failed: client connection to 127.25.109.130:38197: connect: Connection refused (error 111)
I20240502 02:01:32.799646  8205 leader_election.cc:302] T 5be45a68cff24a75a03b2fcd4b7851f9 P 436822a397664c54b8d1b9eb3bc6cff4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 436822a397664c54b8d1b9eb3bc6cff4; no voters: 218bc8c1f9ce46059898fc738179adf9, a2c90ee440b44b708bccc2e2d5385cc0
I20240502 02:01:32.800217 28953 raft_consensus.cc:2726] T 5be45a68cff24a75a03b2fcd4b7851f9 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 02:01:33.190330 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 6385
I20240502 02:01:33.213001 28953 raft_consensus.cc:482] T 086509f1ee0a4a81b2f8194b50dbe159 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2c90ee440b44b708bccc2e2d5385cc0)
I20240502 02:01:33.213245 28953 raft_consensus.cc:504] T 086509f1ee0a4a81b2f8194b50dbe159 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 02:01:33.214156 28953 leader_election.cc:288] T 086509f1ee0a4a81b2f8194b50dbe159 P 436822a397664c54b8d1b9eb3bc6cff4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197), 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359)
W20240502 02:01:33.215449  8203 leader_election.cc:334] T 086509f1ee0a4a81b2f8194b50dbe159 P 436822a397664c54b8d1b9eb3bc6cff4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:01:33.215649  8205 leader_election.cc:334] T 086509f1ee0a4a81b2f8194b50dbe159 P 436822a397664c54b8d1b9eb3bc6cff4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197): Network error: Client connection negotiation failed: client connection to 127.25.109.130:38197: connect: Connection refused (error 111)
I20240502 02:01:33.215790  8205 leader_election.cc:302] T 086509f1ee0a4a81b2f8194b50dbe159 P 436822a397664c54b8d1b9eb3bc6cff4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 436822a397664c54b8d1b9eb3bc6cff4; no voters: 6d34e50fb5dd4a1db58013d4863b8992, a2c90ee440b44b708bccc2e2d5385cc0
I20240502 02:01:33.216090 28953 raft_consensus.cc:2726] T 086509f1ee0a4a81b2f8194b50dbe159 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 02:01:33.245385  8202 connection.cc:535] server connection from 127.25.109.132:38165 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 02:01:33.245785  5907 connection.cc:535] server connection from 127.25.109.132:39769 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 02:01:33.245918 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 7918
I20240502 02:01:33.258462  8038 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 02:01:33.326200 28953 raft_consensus.cc:482] T 99e74a49b42745888058db67049dc682 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 218bc8c1f9ce46059898fc738179adf9)
I20240502 02:01:33.326408 28953 raft_consensus.cc:504] T 99e74a49b42745888058db67049dc682 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 02:01:33.326833 29889 raft_consensus.cc:482] T a398da35f18c4df394db82c5bf89aeb1 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 218bc8c1f9ce46059898fc738179adf9)
I20240502 02:01:33.327003 29889 raft_consensus.cc:504] T a398da35f18c4df394db82c5bf89aeb1 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 02:01:33.327375 28953 leader_election.cc:288] T 99e74a49b42745888058db67049dc682 P 436822a397664c54b8d1b9eb3bc6cff4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227), 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867)
I20240502 02:01:33.328289 29889 leader_election.cc:288] T a398da35f18c4df394db82c5bf89aeb1 P 436822a397664c54b8d1b9eb3bc6cff4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227), 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245)
W20240502 02:01:33.328976  8204 leader_election.cc:334] T 99e74a49b42745888058db67049dc682 P 436822a397664c54b8d1b9eb3bc6cff4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227): Network error: Client connection negotiation failed: client connection to 127.25.109.129:36227: connect: Connection refused (error 111)
W20240502 02:01:33.329140  8202 leader_election.cc:334] T 99e74a49b42745888058db67049dc682 P 436822a397664c54b8d1b9eb3bc6cff4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867): Network error: Client connection negotiation failed: client connection to 127.25.109.132:36867: connect: Connection refused (error 111)
I20240502 02:01:33.329298  8202 leader_election.cc:302] T 99e74a49b42745888058db67049dc682 P 436822a397664c54b8d1b9eb3bc6cff4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 436822a397664c54b8d1b9eb3bc6cff4; no voters: 07cdb213c77c4ffda1f999b66b725399, 218bc8c1f9ce46059898fc738179adf9
I20240502 02:01:33.329588 29889 raft_consensus.cc:2726] T 99e74a49b42745888058db67049dc682 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 02:01:33.330493  8204 leader_election.cc:334] T a398da35f18c4df394db82c5bf89aeb1 P 436822a397664c54b8d1b9eb3bc6cff4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227): Network error: Client connection negotiation failed: client connection to 127.25.109.129:36227: connect: Connection refused (error 111)
W20240502 02:01:33.330683  8204 leader_election.cc:334] T a398da35f18c4df394db82c5bf89aeb1 P 436822a397664c54b8d1b9eb3bc6cff4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245): Network error: Client connection negotiation failed: client connection to 127.25.109.131:44245: connect: Connection refused (error 111)
I20240502 02:01:33.330776  8204 leader_election.cc:302] T a398da35f18c4df394db82c5bf89aeb1 P 436822a397664c54b8d1b9eb3bc6cff4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 436822a397664c54b8d1b9eb3bc6cff4; no voters: 218bc8c1f9ce46059898fc738179adf9, 94ffd7aaf85b4e17881241b8396fa2f3
I20240502 02:01:33.331005 29889 raft_consensus.cc:2726] T a398da35f18c4df394db82c5bf89aeb1 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 02:01:33.437815 29889 raft_consensus.cc:482] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 94ffd7aaf85b4e17881241b8396fa2f3)
I20240502 02:01:33.438015 29889 raft_consensus.cc:504] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 02:01:33.438732 29889 leader_election.cc:288] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245), a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197)
W20240502 02:01:33.440186  8205 leader_election.cc:334] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197): Network error: Client connection negotiation failed: client connection to 127.25.109.130:38197: connect: Connection refused (error 111)
W20240502 02:01:33.440339  8204 leader_election.cc:334] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245): Network error: Client connection negotiation failed: client connection to 127.25.109.131:44245: connect: Connection refused (error 111)
I20240502 02:01:33.440446  8204 leader_election.cc:302] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 436822a397664c54b8d1b9eb3bc6cff4; no voters: 94ffd7aaf85b4e17881241b8396fa2f3, a2c90ee440b44b708bccc2e2d5385cc0
I20240502 02:01:33.440732 29889 raft_consensus.cc:2726] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 02:01:33.611325 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 7918
I20240502 02:01:33.663390 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 8185
I20240502 02:01:33.669574  8308 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 02:01:34.016958 28953 raft_consensus.cc:482] T 73ed91df7ff04846991c95b30a86ea7e P 436822a397664c54b8d1b9eb3bc6cff4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a2c90ee440b44b708bccc2e2d5385cc0)
I20240502 02:01:34.017336 28953 raft_consensus.cc:504] T 73ed91df7ff04846991c95b30a86ea7e P 436822a397664c54b8d1b9eb3bc6cff4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 02:01:34.018111 28953 leader_election.cc:288] T 73ed91df7ff04846991c95b30a86ea7e P 436822a397664c54b8d1b9eb3bc6cff4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197), 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359)
I20240502 02:01:34.020795 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 8185
I20240502 02:01:34.075376 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 5890
I20240502 02:01:34.081307  5949 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 02:01:34.454972 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 5890
2024-05-02T02:01:34Z chronyd exiting
I20240502 02:01:34.529536 26038 test_util.cc:175] -----------------------------------------------
I20240502 02:01:34.529665 26038 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0

Note: This is test shard 3 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from IgnoredTserverGoesDownDuringRebalancingTest
[ RUN      ] IgnoredTserverGoesDownDuringRebalancingTest.TserverDown
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 01:52:26.352862 26038 test_util.cc:268] Using random seed: -622719827
I20240502 01:52:26.365300 26038 ts_itest-base.cc:112] Starting cluster with:
I20240502 01:52:26.365402 26038 ts_itest-base.cc:113] --------------
I20240502 01:52:26.365446 26038 ts_itest-base.cc:114] 8 tablet servers
I20240502 01:52:26.365512 26038 ts_itest-base.cc:115] 3 replicas per TS
I20240502 01:52:26.365576 26038 ts_itest-base.cc:116] --------------
2024-05-02T01:52:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T01:52:26Z Disabled control of system clock
I20240502 01:52:26.390623 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.190:44417
--webserver_interface=127.25.109.190
--webserver_port=0
--builtin_ntp_servers=127.25.109.148:34611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.190:44417
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 01:52:26.709872 26050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:26.710227 26050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:26.710356 26050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:26.718035 26050 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:26.718199 26050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 01:52:26.718283 26050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:26.718364 26050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 01:52:26.718425 26050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 01:52:26.727843 26050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:34611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.25.109.190:44417
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.190:44417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.25.109.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:26.729275 26050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:26.731106 26050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:26.742745 26055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:26.743330 26056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:26.743382 26058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:26.743963 26050 server_base.cc:1034] running on GCE node
I20240502 01:52:26.744786 26050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:26.746281 26050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:26.747498 26050 hybrid_clock.cc:648] HybridClock initialized: now 1714614746747442 us; error 49 us; skew 500 ppm
I20240502 01:52:26.747929 26050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:26.750281 26050 webserver.cc:460] Webserver started at http://127.25.109.190:39109/ using document root <none> and password file <none>
I20240502 01:52:26.751067 26050 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:26.751186 26050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:26.751510 26050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:26.754302 26050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "1d78559e9f0441d594ed99373c093d30"
format_stamp: "Formatted at 2024-05-02 01:52:26 on dist-test-slave-kj4b"
I20240502 01:52:26.755168 26050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "1d78559e9f0441d594ed99373c093d30"
format_stamp: "Formatted at 2024-05-02 01:52:26 on dist-test-slave-kj4b"
I20240502 01:52:26.760708 26050 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.001s
I20240502 01:52:26.764359 26064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:26.765779 26050 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.003s
I20240502 01:52:26.765965 26050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal
uuid: "1d78559e9f0441d594ed99373c093d30"
format_stamp: "Formatted at 2024-05-02 01:52:26 on dist-test-slave-kj4b"
I20240502 01:52:26.766160 26050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:26.785384 26050 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:52:26.786279 26050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:26.786571 26050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:26.804021 26050 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.190:44417
I20240502 01:52:26.804062 26115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.190:44417 every 8 connection(s)
I20240502 01:52:26.805423 26050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 01:52:26.806617 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 26050
I20240502 01:52:26.806859 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 01:52:26.809417 26116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:26.824559 26116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d78559e9f0441d594ed99373c093d30: Bootstrap starting.
I20240502 01:52:26.827841 26116 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d78559e9f0441d594ed99373c093d30: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:26.829006 26116 log.cc:824] T 00000000000000000000000000000000 P 1d78559e9f0441d594ed99373c093d30: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:26.831765 26116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d78559e9f0441d594ed99373c093d30: No bootstrap required, opened a new log
I20240502 01:52:26.839005 26116 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d78559e9f0441d594ed99373c093d30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d78559e9f0441d594ed99373c093d30" member_type: VOTER last_known_addr { host: "127.25.109.190" port: 44417 } }
I20240502 01:52:26.839403 26116 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d78559e9f0441d594ed99373c093d30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:26.839509 26116 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d78559e9f0441d594ed99373c093d30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d78559e9f0441d594ed99373c093d30, State: Initialized, Role: FOLLOWER
I20240502 01:52:26.840152 26116 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d78559e9f0441d594ed99373c093d30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d78559e9f0441d594ed99373c093d30" member_type: VOTER last_known_addr { host: "127.25.109.190" port: 44417 } }
I20240502 01:52:26.840371 26116 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d78559e9f0441d594ed99373c093d30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:26.840493 26116 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d78559e9f0441d594ed99373c093d30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:26.840641 26116 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d78559e9f0441d594ed99373c093d30 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:26.842805 26116 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d78559e9f0441d594ed99373c093d30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d78559e9f0441d594ed99373c093d30" member_type: VOTER last_known_addr { host: "127.25.109.190" port: 44417 } }
I20240502 01:52:26.843405 26116 leader_election.cc:302] T 00000000000000000000000000000000 P 1d78559e9f0441d594ed99373c093d30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d78559e9f0441d594ed99373c093d30; no voters: 
I20240502 01:52:26.844213 26116 leader_election.cc:288] T 00000000000000000000000000000000 P 1d78559e9f0441d594ed99373c093d30 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 01:52:26.844364 26121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d78559e9f0441d594ed99373c093d30 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:26.845008 26121 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d78559e9f0441d594ed99373c093d30 [term 1 LEADER]: Becoming Leader. State: Replica: 1d78559e9f0441d594ed99373c093d30, State: Running, Role: LEADER
I20240502 01:52:26.845678 26121 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d78559e9f0441d594ed99373c093d30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d78559e9f0441d594ed99373c093d30" member_type: VOTER last_known_addr { host: "127.25.109.190" port: 44417 } }
I20240502 01:52:26.846804 26116 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d78559e9f0441d594ed99373c093d30 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:52:26.849036 26123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d78559e9f0441d594ed99373c093d30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d78559e9f0441d594ed99373c093d30. Latest consensus state: current_term: 1 leader_uuid: "1d78559e9f0441d594ed99373c093d30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d78559e9f0441d594ed99373c093d30" member_type: VOTER last_known_addr { host: "127.25.109.190" port: 44417 } } }
I20240502 01:52:26.849001 26122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d78559e9f0441d594ed99373c093d30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d78559e9f0441d594ed99373c093d30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d78559e9f0441d594ed99373c093d30" member_type: VOTER last_known_addr { host: "127.25.109.190" port: 44417 } } }
I20240502 01:52:26.849431 26122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d78559e9f0441d594ed99373c093d30 [sys.catalog]: This master's current role is: LEADER
I20240502 01:52:26.849427 26123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d78559e9f0441d594ed99373c093d30 [sys.catalog]: This master's current role is: LEADER
I20240502 01:52:26.852494 26127 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:52:26.858296 26127 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:52:26.868108 26127 catalog_manager.cc:1347] Generated new cluster ID: a868f4ee9e5547bbb67fd4649a8f91eb
I20240502 01:52:26.868248 26127 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:52:26.884068 26127 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:52:26.885360 26127 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:52:26.895659 26127 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 1d78559e9f0441d594ed99373c093d30: Generated new TSK 0
I20240502 01:52:26.896538 26127 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:52:26.904006 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.129:0
--local_ip_for_outbound_sockets=127.25.109.129
--webserver_interface=127.25.109.129
--webserver_port=0
--tserver_master_addrs=127.25.109.190:44417
--builtin_ntp_servers=127.25.109.148:34611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240502 01:52:27.236449 26140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:27.236765 26140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:27.236919 26140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:27.244586 26140 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:27.244720 26140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:27.244882 26140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.129
I20240502 01:52:27.254426 26140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:34611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.25.109.129
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.25.109.190:44417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.129
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:27.255771 26140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:27.257560 26140 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:27.269579 26146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:27.270069 26145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:27.270324 26148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:27.271474 26140 server_base.cc:1034] running on GCE node
I20240502 01:52:27.272156 26140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:27.273151 26140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:27.274380 26140 hybrid_clock.cc:648] HybridClock initialized: now 1714614747274330 us; error 47 us; skew 500 ppm
I20240502 01:52:27.274724 26140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:27.277102 26140 webserver.cc:460] Webserver started at http://127.25.109.129:41535/ using document root <none> and password file <none>
I20240502 01:52:27.277880 26140 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:27.277999 26140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:27.278319 26140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:27.281126 26140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce"
format_stamp: "Formatted at 2024-05-02 01:52:27 on dist-test-slave-kj4b"
I20240502 01:52:27.281942 26140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce"
format_stamp: "Formatted at 2024-05-02 01:52:27 on dist-test-slave-kj4b"
I20240502 01:52:27.287509 26140 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20240502 01:52:27.291280 26154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:27.292745 26140 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 01:52:27.292941 26140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce"
format_stamp: "Formatted at 2024-05-02 01:52:27 on dist-test-slave-kj4b"
I20240502 01:52:27.293155 26140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:27.313802 26140 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:52:27.314788 26140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:27.315068 26140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:27.316080 26140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:27.317632 26140 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:27.317730 26140 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:27.317850 26140 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:27.317921 26140 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:27.348948 26140 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.129:44541
I20240502 01:52:27.349004 26269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.129:44541 every 8 connection(s)
I20240502 01:52:27.350423 26140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 01:52:27.350968 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 26140
I20240502 01:52:27.351195 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 01:52:27.355360 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.130:0
--local_ip_for_outbound_sockets=127.25.109.130
--webserver_interface=127.25.109.130
--webserver_port=0
--tserver_master_addrs=127.25.109.190:44417
--builtin_ntp_servers=127.25.109.148:34611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 01:52:27.365002 26270 heartbeater.cc:347] Connected to a master server at 127.25.109.190:44417
I20240502 01:52:27.365387 26270 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:27.366356 26270 heartbeater.cc:511] Master 127.25.109.190:44417 requested a full tablet report, sending...
I20240502 01:52:27.368763 26081 ts_manager.cc:194] Registered new tserver with Master: f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541)
I20240502 01:52:27.370353 26081 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.129:35551
W20240502 01:52:27.694442 26274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:27.694792 26274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:27.694947 26274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:27.702673 26274 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:27.702837 26274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:27.703003 26274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.130
I20240502 01:52:27.712531 26274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:34611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.25.109.130
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.25.109.190:44417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.130
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:27.713960 26274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:27.715915 26274 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:27.729190 26282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:27.729168 26279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:27.729761 26280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:27.730235 26274 server_base.cc:1034] running on GCE node
I20240502 01:52:27.730911 26274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:27.731977 26274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:27.733196 26274 hybrid_clock.cc:648] HybridClock initialized: now 1714614747733138 us; error 82 us; skew 500 ppm
I20240502 01:52:27.733543 26274 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:27.735901 26274 webserver.cc:460] Webserver started at http://127.25.109.130:46407/ using document root <none> and password file <none>
I20240502 01:52:27.736672 26274 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:27.736799 26274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:27.737123 26274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:27.739748 26274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "868e6e4358f4475bb45967b05eccb09f"
format_stamp: "Formatted at 2024-05-02 01:52:27 on dist-test-slave-kj4b"
I20240502 01:52:27.740569 26274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "868e6e4358f4475bb45967b05eccb09f"
format_stamp: "Formatted at 2024-05-02 01:52:27 on dist-test-slave-kj4b"
I20240502 01:52:27.745788 26274 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20240502 01:52:27.749418 26288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:27.750892 26274 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20240502 01:52:27.751072 26274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "868e6e4358f4475bb45967b05eccb09f"
format_stamp: "Formatted at 2024-05-02 01:52:27 on dist-test-slave-kj4b"
I20240502 01:52:27.751304 26274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:27.776150 26274 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:52:27.777093 26274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:27.777388 26274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:27.778430 26274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:27.780128 26274 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:27.780233 26274 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:27.780351 26274 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:27.780422 26274 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:27.812887 26274 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.130:45571
I20240502 01:52:27.812924 26401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.130:45571 every 8 connection(s)
I20240502 01:52:27.814278 26274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 01:52:27.824328 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 26274
I20240502 01:52:27.824498 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 01:52:27.828146 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.131:0
--local_ip_for_outbound_sockets=127.25.109.131
--webserver_interface=127.25.109.131
--webserver_port=0
--tserver_master_addrs=127.25.109.190:44417
--builtin_ntp_servers=127.25.109.148:34611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 01:52:27.828718 26402 heartbeater.cc:347] Connected to a master server at 127.25.109.190:44417
I20240502 01:52:27.829093 26402 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:27.830230 26402 heartbeater.cc:511] Master 127.25.109.190:44417 requested a full tablet report, sending...
I20240502 01:52:27.831923 26081 ts_manager.cc:194] Registered new tserver with Master: 868e6e4358f4475bb45967b05eccb09f (127.25.109.130:45571)
I20240502 01:52:27.832998 26081 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.130:55377
W20240502 01:52:28.173272 26405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:28.173600 26405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:28.173745 26405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:28.181308 26405 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:28.181453 26405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:28.181617 26405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.131
I20240502 01:52:28.190855 26405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:34611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.25.109.131
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.25.109.190:44417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.131
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:28.192294 26405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:28.194077 26405 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:28.206511 26412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:28.207037 26414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:28.207597 26411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:28.208094 26405 server_base.cc:1034] running on GCE node
I20240502 01:52:28.208678 26405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:28.209625 26405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:28.210811 26405 hybrid_clock.cc:648] HybridClock initialized: now 1714614748210753 us; error 55 us; skew 500 ppm
I20240502 01:52:28.211130 26405 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:28.213404 26405 webserver.cc:460] Webserver started at http://127.25.109.131:45173/ using document root <none> and password file <none>
I20240502 01:52:28.214166 26405 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:28.214273 26405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:28.214583 26405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:28.217386 26405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96"
format_stamp: "Formatted at 2024-05-02 01:52:28 on dist-test-slave-kj4b"
I20240502 01:52:28.218170 26405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96"
format_stamp: "Formatted at 2024-05-02 01:52:28 on dist-test-slave-kj4b"
I20240502 01:52:28.223443 26405 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.001s
I20240502 01:52:28.227377 26420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:28.229139 26405 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
I20240502 01:52:28.229351 26405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96"
format_stamp: "Formatted at 2024-05-02 01:52:28 on dist-test-slave-kj4b"
I20240502 01:52:28.229583 26405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:28.246681 26405 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:52:28.247603 26405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:28.247942 26405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:28.248967 26405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:28.250507 26405 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:28.250610 26405 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:28.250720 26405 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:28.250787 26405 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:28.281205 26405 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.131:38693
I20240502 01:52:28.281258 26534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.131:38693 every 8 connection(s)
I20240502 01:52:28.282536 26405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 01:52:28.287496 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 26405
I20240502 01:52:28.287678 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 01:52:28.290724 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.132:0
--local_ip_for_outbound_sockets=127.25.109.132
--webserver_interface=127.25.109.132
--webserver_port=0
--tserver_master_addrs=127.25.109.190:44417
--builtin_ntp_servers=127.25.109.148:34611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 01:52:28.295506 26535 heartbeater.cc:347] Connected to a master server at 127.25.109.190:44417
I20240502 01:52:28.295810 26535 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:28.296700 26535 heartbeater.cc:511] Master 127.25.109.190:44417 requested a full tablet report, sending...
I20240502 01:52:28.298332 26081 ts_manager.cc:194] Registered new tserver with Master: 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131:38693)
I20240502 01:52:28.299417 26081 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.131:60061
I20240502 01:52:28.373459 26270 heartbeater.cc:503] Master 127.25.109.190:44417 was elected leader, sending a full tablet report...
W20240502 01:52:28.617750 26538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:28.618052 26538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:28.618165 26538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:28.625653 26538 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:28.625792 26538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:28.625919 26538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.132
I20240502 01:52:28.635355 26538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:34611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.25.109.132
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.25.109.190:44417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.132
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:28.636737 26538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:28.638489 26538 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:28.650596 26543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:28.651021 26544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:28.651407 26546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:28.651927 26538 server_base.cc:1034] running on GCE node
I20240502 01:52:28.652555 26538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:28.653604 26538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:28.654832 26538 hybrid_clock.cc:648] HybridClock initialized: now 1714614748654772 us; error 60 us; skew 500 ppm
I20240502 01:52:28.655196 26538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:28.657673 26538 webserver.cc:460] Webserver started at http://127.25.109.132:40897/ using document root <none> and password file <none>
I20240502 01:52:28.658483 26538 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:28.658597 26538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:28.658921 26538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:28.661610 26538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "d1e7d0eabafa46f382e19f596e2fc00d"
format_stamp: "Formatted at 2024-05-02 01:52:28 on dist-test-slave-kj4b"
I20240502 01:52:28.662526 26538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "d1e7d0eabafa46f382e19f596e2fc00d"
format_stamp: "Formatted at 2024-05-02 01:52:28 on dist-test-slave-kj4b"
I20240502 01:52:28.667765 26538 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.001s
I20240502 01:52:28.671254 26552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:28.672717 26538 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 01:52:28.672889 26538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "d1e7d0eabafa46f382e19f596e2fc00d"
format_stamp: "Formatted at 2024-05-02 01:52:28 on dist-test-slave-kj4b"
I20240502 01:52:28.673096 26538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:28.692837 26538 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:52:28.693720 26538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:28.693994 26538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:28.694949 26538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:28.696558 26538 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:28.696655 26538 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:28.696763 26538 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:28.696831 26538 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:28.726423 26538 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.132:37695
I20240502 01:52:28.726440 26667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.132:37695 every 8 connection(s)
I20240502 01:52:28.727821 26538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 01:52:28.737226 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 26538
I20240502 01:52:28.737421 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 01:52:28.740270 26668 heartbeater.cc:347] Connected to a master server at 127.25.109.190:44417
I20240502 01:52:28.740614 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.133:0
--local_ip_for_outbound_sockets=127.25.109.133
--webserver_interface=127.25.109.133
--webserver_port=0
--tserver_master_addrs=127.25.109.190:44417
--builtin_ntp_servers=127.25.109.148:34611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 01:52:28.740677 26668 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:28.741663 26668 heartbeater.cc:511] Master 127.25.109.190:44417 requested a full tablet report, sending...
I20240502 01:52:28.743203 26081 ts_manager.cc:194] Registered new tserver with Master: d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132:37695)
I20240502 01:52:28.744252 26081 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.132:42635
I20240502 01:52:28.836436 26402 heartbeater.cc:503] Master 127.25.109.190:44417 was elected leader, sending a full tablet report...
W20240502 01:52:29.070044 26671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:29.070376 26671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:29.070524 26671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:29.078254 26671 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:29.078399 26671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:29.078555 26671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.133
I20240502 01:52:29.087843 26671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:34611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.25.109.133
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.25.109.190:44417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.133
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:29.089301 26671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:29.091125 26671 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:29.103893 26679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:29.103825 26677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:29.104482 26676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:29.104915 26671 server_base.cc:1034] running on GCE node
I20240502 01:52:29.105540 26671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:29.106501 26671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:29.107718 26671 hybrid_clock.cc:648] HybridClock initialized: now 1714614749107654 us; error 47 us; skew 500 ppm
I20240502 01:52:29.108083 26671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:29.110463 26671 webserver.cc:460] Webserver started at http://127.25.109.133:38707/ using document root <none> and password file <none>
I20240502 01:52:29.111233 26671 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:29.111343 26671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:29.111673 26671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:29.114423 26671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "ca931e9360d44079b6059b8d9e7a6d20"
format_stamp: "Formatted at 2024-05-02 01:52:29 on dist-test-slave-kj4b"
I20240502 01:52:29.115180 26671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "ca931e9360d44079b6059b8d9e7a6d20"
format_stamp: "Formatted at 2024-05-02 01:52:29 on dist-test-slave-kj4b"
I20240502 01:52:29.120828 26671 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.008s	sys 0.000s
I20240502 01:52:29.124342 26685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:29.125766 26671 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.002s
I20240502 01:52:29.125948 26671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "ca931e9360d44079b6059b8d9e7a6d20"
format_stamp: "Formatted at 2024-05-02 01:52:29 on dist-test-slave-kj4b"
I20240502 01:52:29.126147 26671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:29.140589 26671 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:52:29.141498 26671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:29.141774 26671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:29.142786 26671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:29.144455 26671 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:29.144556 26671 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:29.144670 26671 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:29.144738 26671 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:29.175060 26671 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.133:33081
I20240502 01:52:29.175107 26798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.133:33081 every 8 connection(s)
I20240502 01:52:29.176460 26671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:52:29.177439 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 26671
I20240502 01:52:29.177654 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 01:52:29.181341 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.134:0
--local_ip_for_outbound_sockets=127.25.109.134
--webserver_interface=127.25.109.134
--webserver_port=0
--tserver_master_addrs=127.25.109.190:44417
--builtin_ntp_servers=127.25.109.148:34611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 01:52:29.189847 26799 heartbeater.cc:347] Connected to a master server at 127.25.109.190:44417
I20240502 01:52:29.190151 26799 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:29.191044 26799 heartbeater.cc:511] Master 127.25.109.190:44417 requested a full tablet report, sending...
I20240502 01:52:29.192597 26081 ts_manager.cc:194] Registered new tserver with Master: ca931e9360d44079b6059b8d9e7a6d20 (127.25.109.133:33081)
I20240502 01:52:29.193683 26081 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.133:47445
I20240502 01:52:29.302312 26535 heartbeater.cc:503] Master 127.25.109.190:44417 was elected leader, sending a full tablet report...
W20240502 01:52:29.519590 26802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:29.519935 26802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:29.520043 26802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:29.527624 26802 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:29.527774 26802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:29.527932 26802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.134
I20240502 01:52:29.537652 26802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:34611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.25.109.134
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.25.109.190:44417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.134
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:29.539021 26802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:29.540818 26802 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:29.553143 26810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:29.553629 26808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:29.553920 26807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:29.554240 26802 server_base.cc:1034] running on GCE node
I20240502 01:52:29.554854 26802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:29.555953 26802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:29.557175 26802 hybrid_clock.cc:648] HybridClock initialized: now 1714614749557108 us; error 57 us; skew 500 ppm
I20240502 01:52:29.557511 26802 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:29.559826 26802 webserver.cc:460] Webserver started at http://127.25.109.134:44899/ using document root <none> and password file <none>
I20240502 01:52:29.560640 26802 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:29.560743 26802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:29.561050 26802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:29.563733 26802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "6ecf0a12e15f45a3bfa042f88c100a5e"
format_stamp: "Formatted at 2024-05-02 01:52:29 on dist-test-slave-kj4b"
I20240502 01:52:29.564554 26802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "6ecf0a12e15f45a3bfa042f88c100a5e"
format_stamp: "Formatted at 2024-05-02 01:52:29 on dist-test-slave-kj4b"
I20240502 01:52:29.570278 26802 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.002s	sys 0.003s
I20240502 01:52:29.574256 26816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:29.575804 26802 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:52:29.576062 26802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "6ecf0a12e15f45a3bfa042f88c100a5e"
format_stamp: "Formatted at 2024-05-02 01:52:29 on dist-test-slave-kj4b"
I20240502 01:52:29.576294 26802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:29.601114 26802 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:52:29.602020 26802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:29.602296 26802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:29.603264 26802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:29.604990 26802 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:29.605079 26802 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:29.605187 26802 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:29.605263 26802 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:29.634682 26802 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.134:43811
I20240502 01:52:29.634753 26929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.134:43811 every 8 connection(s)
I20240502 01:52:29.636067 26802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 01:52:29.639458 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 26802
I20240502 01:52:29.639624 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 01:52:29.642722 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.135:0
--local_ip_for_outbound_sockets=127.25.109.135
--webserver_interface=127.25.109.135
--webserver_port=0
--tserver_master_addrs=127.25.109.190:44417
--builtin_ntp_servers=127.25.109.148:34611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 01:52:29.649009 26930 heartbeater.cc:347] Connected to a master server at 127.25.109.190:44417
I20240502 01:52:29.649390 26930 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:29.650501 26930 heartbeater.cc:511] Master 127.25.109.190:44417 requested a full tablet report, sending...
I20240502 01:52:29.652369 26081 ts_manager.cc:194] Registered new tserver with Master: 6ecf0a12e15f45a3bfa042f88c100a5e (127.25.109.134:43811)
I20240502 01:52:29.653210 26081 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.134:38161
I20240502 01:52:29.746932 26668 heartbeater.cc:503] Master 127.25.109.190:44417 was elected leader, sending a full tablet report...
W20240502 01:52:29.974515 26934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:29.974819 26934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:29.974967 26934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:29.982748 26934 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:29.982887 26934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:29.983050 26934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.135
I20240502 01:52:29.992446 26934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:34611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.25.109.135
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.25.109.190:44417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.135
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:29.993794 26934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:29.995548 26934 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:30.007782 26939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:30.008230 26940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:30.009047 26942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:30.009598 26934 server_base.cc:1034] running on GCE node
I20240502 01:52:30.010232 26934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:30.011217 26934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:30.012423 26934 hybrid_clock.cc:648] HybridClock initialized: now 1714614750012357 us; error 53 us; skew 500 ppm
I20240502 01:52:30.012797 26934 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:30.019254 26934 webserver.cc:460] Webserver started at http://127.25.109.135:39507/ using document root <none> and password file <none>
I20240502 01:52:30.020174 26934 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:30.020298 26934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:30.020619 26934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:30.023237 26934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "aed397efb8844970b606bca15a77cb59"
format_stamp: "Formatted at 2024-05-02 01:52:30 on dist-test-slave-kj4b"
I20240502 01:52:30.024041 26934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "aed397efb8844970b606bca15a77cb59"
format_stamp: "Formatted at 2024-05-02 01:52:30 on dist-test-slave-kj4b"
I20240502 01:52:30.029168 26934 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20240502 01:52:30.032615 26948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:30.033965 26934 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240502 01:52:30.034127 26934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "aed397efb8844970b606bca15a77cb59"
format_stamp: "Formatted at 2024-05-02 01:52:30 on dist-test-slave-kj4b"
I20240502 01:52:30.034332 26934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:30.047122 26934 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:52:30.048189 26934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:30.048477 26934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:30.049465 26934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:30.051045 26934 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:30.051151 26934 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:30.051270 26934 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:30.051340 26934 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:30.082556 26934 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.135:39905
I20240502 01:52:30.082613 27063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.135:39905 every 8 connection(s)
I20240502 01:52:30.083993 26934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 01:52:30.091069 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 26934
I20240502 01:52:30.091300 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 01:52:30.095054 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.136:0
--local_ip_for_outbound_sockets=127.25.109.136
--webserver_interface=127.25.109.136
--webserver_port=0
--tserver_master_addrs=127.25.109.190:44417
--builtin_ntp_servers=127.25.109.148:34611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 01:52:30.098798 27064 heartbeater.cc:347] Connected to a master server at 127.25.109.190:44417
I20240502 01:52:30.099150 27064 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:30.100155 27064 heartbeater.cc:511] Master 127.25.109.190:44417 requested a full tablet report, sending...
I20240502 01:52:30.101929 26081 ts_manager.cc:194] Registered new tserver with Master: aed397efb8844970b606bca15a77cb59 (127.25.109.135:39905)
I20240502 01:52:30.102782 26081 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.135:37537
I20240502 01:52:30.196550 26799 heartbeater.cc:503] Master 127.25.109.190:44417 was elected leader, sending a full tablet report...
W20240502 01:52:30.452477 27067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:30.452800 27067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:30.452915 27067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:30.462168 27067 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:30.462383 27067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:30.462630 27067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.136
I20240502 01:52:30.475088 27067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:34611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.25.109.136
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.25.109.190:44417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.136
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:30.476665 27067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:30.478910 27067 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:30.492282 27072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:30.492570 27075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:30.493038 27073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:30.493474 27067 server_base.cc:1034] running on GCE node
I20240502 01:52:30.494114 27067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:30.495110 27067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:30.496323 27067 hybrid_clock.cc:648] HybridClock initialized: now 1714614750496266 us; error 50 us; skew 500 ppm
I20240502 01:52:30.496683 27067 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:30.499403 27067 webserver.cc:460] Webserver started at http://127.25.109.136:33819/ using document root <none> and password file <none>
I20240502 01:52:30.500254 27067 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:30.500383 27067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:30.500711 27067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:30.503813 27067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "c292d733f9584891b46f52aafee403fc"
format_stamp: "Formatted at 2024-05-02 01:52:30 on dist-test-slave-kj4b"
I20240502 01:52:30.504909 27067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "c292d733f9584891b46f52aafee403fc"
format_stamp: "Formatted at 2024-05-02 01:52:30 on dist-test-slave-kj4b"
I20240502 01:52:30.511206 27067 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.006s	sys 0.000s
I20240502 01:52:30.515408 27081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:30.516892 27067 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.002s
I20240502 01:52:30.517086 27067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "c292d733f9584891b46f52aafee403fc"
format_stamp: "Formatted at 2024-05-02 01:52:30 on dist-test-slave-kj4b"
I20240502 01:52:30.517305 27067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:30.532810 27067 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:52:30.533766 27067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:30.534067 27067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:30.535059 27067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:30.536803 27067 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:30.536974 27067 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:30.537122 27067 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:30.537199 27067 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:30.578656 27067 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.136:38319
I20240502 01:52:30.578795 27194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.136:38319 every 8 connection(s)
I20240502 01:52:30.580123 27067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 01:52:30.583178 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 27067
I20240502 01:52:30.583384 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 01:52:30.594638 27195 heartbeater.cc:347] Connected to a master server at 127.25.109.190:44417
I20240502 01:52:30.594990 27195 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:30.595979 27195 heartbeater.cc:511] Master 127.25.109.190:44417 requested a full tablet report, sending...
I20240502 01:52:30.597656 26081 ts_manager.cc:194] Registered new tserver with Master: c292d733f9584891b46f52aafee403fc (127.25.109.136:38319)
I20240502 01:52:30.598663 26081 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.136:44111
I20240502 01:52:30.601384 26038 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 01:52:30.622727 26038 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 01:52:30.622974 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 26671
I20240502 01:52:30.632122 26794 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:52:30.656847 26930 heartbeater.cc:503] Master 127.25.109.190:44417 was elected leader, sending a full tablet report...
I20240502 01:52:30.713837 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 26671
I20240502 01:52:30.728170 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 26802
I20240502 01:52:30.736946 26925 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:52:30.820605 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 26802
I20240502 01:52:30.839572 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 26934
I20240502 01:52:30.848147 27059 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:52:30.928853 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 26934
I20240502 01:52:30.942770 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 27067
I20240502 01:52:30.951668 27190 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:52:31.032415 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 27067
I20240502 01:52:34.800060 26081 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57636:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:52:34.845356 26469 tablet_service.cc:1467] Processing CreateTablet for tablet 895cfddfe1804d34a80b8a52e00efce2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=45682f0e9b4f45c3b3e4cb8de40b23bd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:34.845527 26601 tablet_service.cc:1467] Processing CreateTablet for tablet 895cfddfe1804d34a80b8a52e00efce2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=45682f0e9b4f45c3b3e4cb8de40b23bd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:34.846666 26204 tablet_service.cc:1467] Processing CreateTablet for tablet c32bdf0d644e499eaa2f755fab9abb77 (DEFAULT_TABLE table=rebalance_test_table_0 [id=45682f0e9b4f45c3b3e4cb8de40b23bd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:34.847121 26601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 895cfddfe1804d34a80b8a52e00efce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:34.846355 26203 tablet_service.cc:1467] Processing CreateTablet for tablet dc44aad1f4ab4469a29d280a11268be0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=45682f0e9b4f45c3b3e4cb8de40b23bd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:34.847812 26204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c32bdf0d644e499eaa2f755fab9abb77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:34.848578 26203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc44aad1f4ab4469a29d280a11268be0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:34.848275 26468 tablet_service.cc:1467] Processing CreateTablet for tablet c32bdf0d644e499eaa2f755fab9abb77 (DEFAULT_TABLE table=rebalance_test_table_0 [id=45682f0e9b4f45c3b3e4cb8de40b23bd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:34.851040 26468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c32bdf0d644e499eaa2f755fab9abb77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:34.851222 26600 tablet_service.cc:1467] Processing CreateTablet for tablet dc44aad1f4ab4469a29d280a11268be0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=45682f0e9b4f45c3b3e4cb8de40b23bd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:34.851788 26600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc44aad1f4ab4469a29d280a11268be0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:34.856236 26336 tablet_service.cc:1467] Processing CreateTablet for tablet 895cfddfe1804d34a80b8a52e00efce2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=45682f0e9b4f45c3b3e4cb8de40b23bd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:34.856487 26335 tablet_service.cc:1467] Processing CreateTablet for tablet c32bdf0d644e499eaa2f755fab9abb77 (DEFAULT_TABLE table=rebalance_test_table_0 [id=45682f0e9b4f45c3b3e4cb8de40b23bd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:34.857302 26336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 895cfddfe1804d34a80b8a52e00efce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:34.857447 26469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 895cfddfe1804d34a80b8a52e00efce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:34.860321 26335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c32bdf0d644e499eaa2f755fab9abb77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:34.855260 26334 tablet_service.cc:1467] Processing CreateTablet for tablet dc44aad1f4ab4469a29d280a11268be0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=45682f0e9b4f45c3b3e4cb8de40b23bd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:34.867686 26334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc44aad1f4ab4469a29d280a11268be0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:34.870745 27225 tablet_bootstrap.cc:492] T dc44aad1f4ab4469a29d280a11268be0 P d1e7d0eabafa46f382e19f596e2fc00d: Bootstrap starting.
I20240502 01:52:34.874992 27225 tablet_bootstrap.cc:654] T dc44aad1f4ab4469a29d280a11268be0 P d1e7d0eabafa46f382e19f596e2fc00d: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:34.876004 27226 tablet_bootstrap.cc:492] T dc44aad1f4ab4469a29d280a11268be0 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap starting.
I20240502 01:52:34.876503 27227 tablet_bootstrap.cc:492] T c32bdf0d644e499eaa2f755fab9abb77 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Bootstrap starting.
I20240502 01:52:34.879062 27225 log.cc:824] T dc44aad1f4ab4469a29d280a11268be0 P d1e7d0eabafa46f382e19f596e2fc00d: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:34.880272 27227 tablet_bootstrap.cc:654] T c32bdf0d644e499eaa2f755fab9abb77 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:34.881582 27227 log.cc:824] T c32bdf0d644e499eaa2f755fab9abb77 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:34.881286 27226 tablet_bootstrap.cc:654] T dc44aad1f4ab4469a29d280a11268be0 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:34.883483 27226 log.cc:824] T dc44aad1f4ab4469a29d280a11268be0 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:34.890064 27226 tablet_bootstrap.cc:492] T dc44aad1f4ab4469a29d280a11268be0 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: No bootstrap required, opened a new log
I20240502 01:52:34.890971 27226 ts_tablet_manager.cc:1367] T dc44aad1f4ab4469a29d280a11268be0 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent bootstrapping tablet: real 0.015s	user 0.009s	sys 0.000s
I20240502 01:52:34.891129 27227 tablet_bootstrap.cc:492] T c32bdf0d644e499eaa2f755fab9abb77 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: No bootstrap required, opened a new log
I20240502 01:52:34.891505 27227 ts_tablet_manager.cc:1367] T c32bdf0d644e499eaa2f755fab9abb77 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Time spent bootstrapping tablet: real 0.016s	user 0.007s	sys 0.007s
I20240502 01:52:34.892361 27225 tablet_bootstrap.cc:492] T dc44aad1f4ab4469a29d280a11268be0 P d1e7d0eabafa46f382e19f596e2fc00d: No bootstrap required, opened a new log
I20240502 01:52:34.892699 27225 ts_tablet_manager.cc:1367] T dc44aad1f4ab4469a29d280a11268be0 P d1e7d0eabafa46f382e19f596e2fc00d: Time spent bootstrapping tablet: real 0.022s	user 0.010s	sys 0.000s
I20240502 01:52:34.900741 27228 tablet_bootstrap.cc:492] T c32bdf0d644e499eaa2f755fab9abb77 P 868e6e4358f4475bb45967b05eccb09f: Bootstrap starting.
I20240502 01:52:34.901590 27227 raft_consensus.cc:348] T c32bdf0d644e499eaa2f755fab9abb77 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } }
I20240502 01:52:34.901551 27225 raft_consensus.cc:348] T dc44aad1f4ab4469a29d280a11268be0 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:34.902079 27225 raft_consensus.cc:374] T dc44aad1f4ab4469a29d280a11268be0 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:34.902236 27227 raft_consensus.cc:374] T c32bdf0d644e499eaa2f755fab9abb77 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:34.902256 27225 raft_consensus.cc:729] T dc44aad1f4ab4469a29d280a11268be0 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e7d0eabafa46f382e19f596e2fc00d, State: Initialized, Role: FOLLOWER
I20240502 01:52:34.902395 27227 raft_consensus.cc:729] T c32bdf0d644e499eaa2f755fab9abb77 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c70fbdc8a0b42ad8c09c6dbc842cc96, State: Initialized, Role: FOLLOWER
I20240502 01:52:34.902917 27226 raft_consensus.cc:348] T dc44aad1f4ab4469a29d280a11268be0 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:34.903198 27225 consensus_queue.cc:260] T dc44aad1f4ab4469a29d280a11268be0 P d1e7d0eabafa46f382e19f596e2fc00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:34.903445 27226 raft_consensus.cc:374] T dc44aad1f4ab4469a29d280a11268be0 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:34.903276 27227 consensus_queue.cc:260] T c32bdf0d644e499eaa2f755fab9abb77 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } }
I20240502 01:52:34.903594 27226 raft_consensus.cc:729] T dc44aad1f4ab4469a29d280a11268be0 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f215e8fdc8cb45f9a5eff7cb9e6ccfce, State: Initialized, Role: FOLLOWER
I20240502 01:52:34.904465 27226 consensus_queue.cc:260] T dc44aad1f4ab4469a29d280a11268be0 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:34.904791 27228 tablet_bootstrap.cc:654] T c32bdf0d644e499eaa2f755fab9abb77 P 868e6e4358f4475bb45967b05eccb09f: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:34.906325 27228 log.cc:824] T c32bdf0d644e499eaa2f755fab9abb77 P 868e6e4358f4475bb45967b05eccb09f: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:34.912792 27226 ts_tablet_manager.cc:1398] T dc44aad1f4ab4469a29d280a11268be0 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent starting tablet: real 0.022s	user 0.019s	sys 0.000s
I20240502 01:52:34.922171 27226 tablet_bootstrap.cc:492] T c32bdf0d644e499eaa2f755fab9abb77 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap starting.
I20240502 01:52:34.922894 27236 raft_consensus.cc:482] T dc44aad1f4ab4469a29d280a11268be0 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:34.923254 27236 raft_consensus.cc:504] T dc44aad1f4ab4469a29d280a11268be0 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:34.924036 27226 tablet_bootstrap.cc:654] T c32bdf0d644e499eaa2f755fab9abb77 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:34.932569 27226 tablet_bootstrap.cc:492] T c32bdf0d644e499eaa2f755fab9abb77 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: No bootstrap required, opened a new log
I20240502 01:52:34.932847 27226 ts_tablet_manager.cc:1367] T c32bdf0d644e499eaa2f755fab9abb77 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent bootstrapping tablet: real 0.011s	user 0.002s	sys 0.007s
I20240502 01:52:34.934185 27226 raft_consensus.cc:348] T c32bdf0d644e499eaa2f755fab9abb77 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } }
I20240502 01:52:34.934501 27226 raft_consensus.cc:374] T c32bdf0d644e499eaa2f755fab9abb77 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:34.934646 27226 raft_consensus.cc:729] T c32bdf0d644e499eaa2f755fab9abb77 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f215e8fdc8cb45f9a5eff7cb9e6ccfce, State: Initialized, Role: FOLLOWER
I20240502 01:52:34.935088 27226 consensus_queue.cc:260] T c32bdf0d644e499eaa2f755fab9abb77 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } }
I20240502 01:52:34.936421 27236 leader_election.cc:288] T dc44aad1f4ab4469a29d280a11268be0 P d1e7d0eabafa46f382e19f596e2fc00d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541), 868e6e4358f4475bb45967b05eccb09f (127.25.109.130:45571)
I20240502 01:52:34.937150 27227 ts_tablet_manager.cc:1398] T c32bdf0d644e499eaa2f755fab9abb77 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Time spent starting tablet: real 0.045s	user 0.016s	sys 0.026s
I20240502 01:52:34.938720 27227 tablet_bootstrap.cc:492] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Bootstrap starting.
I20240502 01:52:34.940536 27227 tablet_bootstrap.cc:654] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:34.943675 27226 ts_tablet_manager.cc:1398] T c32bdf0d644e499eaa2f755fab9abb77 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent starting tablet: real 0.011s	user 0.000s	sys 0.002s
I20240502 01:52:34.945230 27225 ts_tablet_manager.cc:1398] T dc44aad1f4ab4469a29d280a11268be0 P d1e7d0eabafa46f382e19f596e2fc00d: Time spent starting tablet: real 0.052s	user 0.005s	sys 0.015s
I20240502 01:52:34.946534 27225 tablet_bootstrap.cc:492] T 895cfddfe1804d34a80b8a52e00efce2 P d1e7d0eabafa46f382e19f596e2fc00d: Bootstrap starting.
I20240502 01:52:34.948510 27225 tablet_bootstrap.cc:654] T 895cfddfe1804d34a80b8a52e00efce2 P d1e7d0eabafa46f382e19f596e2fc00d: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:34.948923 27227 tablet_bootstrap.cc:492] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: No bootstrap required, opened a new log
I20240502 01:52:34.949103 27227 ts_tablet_manager.cc:1367] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Time spent bootstrapping tablet: real 0.010s	user 0.003s	sys 0.000s
I20240502 01:52:34.950204 27227 raft_consensus.cc:348] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:34.950446 27227 raft_consensus.cc:374] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:34.950541 27227 raft_consensus.cc:729] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c70fbdc8a0b42ad8c09c6dbc842cc96, State: Initialized, Role: FOLLOWER
I20240502 01:52:34.950801 27227 consensus_queue.cc:260] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:34.951400 27227 ts_tablet_manager.cc:1398] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:52:34.952759 26224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc44aad1f4ab4469a29d280a11268be0" candidate_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" is_pre_election: true
I20240502 01:52:34.953248 27228 tablet_bootstrap.cc:492] T c32bdf0d644e499eaa2f755fab9abb77 P 868e6e4358f4475bb45967b05eccb09f: No bootstrap required, opened a new log
I20240502 01:52:34.953341 26224 raft_consensus.cc:2454] T dc44aad1f4ab4469a29d280a11268be0 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e7d0eabafa46f382e19f596e2fc00d in term 0.
I20240502 01:52:34.953603 27228 ts_tablet_manager.cc:1367] T c32bdf0d644e499eaa2f755fab9abb77 P 868e6e4358f4475bb45967b05eccb09f: Time spent bootstrapping tablet: real 0.053s	user 0.023s	sys 0.024s
I20240502 01:52:34.954257 26556 leader_election.cc:302] T dc44aad1f4ab4469a29d280a11268be0 P d1e7d0eabafa46f382e19f596e2fc00d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e7d0eabafa46f382e19f596e2fc00d, f215e8fdc8cb45f9a5eff7cb9e6ccfce; no voters: 
I20240502 01:52:34.954846 27236 raft_consensus.cc:2781] T dc44aad1f4ab4469a29d280a11268be0 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:34.955090 27236 raft_consensus.cc:482] T dc44aad1f4ab4469a29d280a11268be0 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:34.955271 27236 raft_consensus.cc:3037] T dc44aad1f4ab4469a29d280a11268be0 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:34.959388 27236 raft_consensus.cc:504] T dc44aad1f4ab4469a29d280a11268be0 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:34.960407 26357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc44aad1f4ab4469a29d280a11268be0" candidate_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "868e6e4358f4475bb45967b05eccb09f" is_pre_election: true
I20240502 01:52:34.961009 26224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc44aad1f4ab4469a29d280a11268be0" candidate_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce"
I20240502 01:52:34.961087 26356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc44aad1f4ab4469a29d280a11268be0" candidate_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "868e6e4358f4475bb45967b05eccb09f"
I20240502 01:52:34.961241 26224 raft_consensus.cc:3037] T dc44aad1f4ab4469a29d280a11268be0 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:34.961638 27236 leader_election.cc:288] T dc44aad1f4ab4469a29d280a11268be0 P d1e7d0eabafa46f382e19f596e2fc00d [CANDIDATE]: Term 1 election: Requested vote from peers f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541), 868e6e4358f4475bb45967b05eccb09f (127.25.109.130:45571)
W20240502 01:52:34.962018 26555 leader_election.cc:341] T dc44aad1f4ab4469a29d280a11268be0 P d1e7d0eabafa46f382e19f596e2fc00d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 868e6e4358f4475bb45967b05eccb09f (127.25.109.130:45571): Illegal state: must be running to vote when last-logged opid is not known
W20240502 01:52:34.962327 26555 leader_election.cc:341] T dc44aad1f4ab4469a29d280a11268be0 P d1e7d0eabafa46f382e19f596e2fc00d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 868e6e4358f4475bb45967b05eccb09f (127.25.109.130:45571): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:52:34.962785 27225 tablet_bootstrap.cc:492] T 895cfddfe1804d34a80b8a52e00efce2 P d1e7d0eabafa46f382e19f596e2fc00d: No bootstrap required, opened a new log
I20240502 01:52:34.962992 27225 ts_tablet_manager.cc:1367] T 895cfddfe1804d34a80b8a52e00efce2 P d1e7d0eabafa46f382e19f596e2fc00d: Time spent bootstrapping tablet: real 0.017s	user 0.008s	sys 0.000s
I20240502 01:52:34.964095 27225 raft_consensus.cc:348] T 895cfddfe1804d34a80b8a52e00efce2 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:34.963992 27228 raft_consensus.cc:348] T c32bdf0d644e499eaa2f755fab9abb77 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } }
I20240502 01:52:34.964344 27225 raft_consensus.cc:374] T 895cfddfe1804d34a80b8a52e00efce2 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:34.964329 26224 raft_consensus.cc:2454] T dc44aad1f4ab4469a29d280a11268be0 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e7d0eabafa46f382e19f596e2fc00d in term 1.
I20240502 01:52:34.964455 27225 raft_consensus.cc:729] T 895cfddfe1804d34a80b8a52e00efce2 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e7d0eabafa46f382e19f596e2fc00d, State: Initialized, Role: FOLLOWER
I20240502 01:52:34.964468 27228 raft_consensus.cc:374] T c32bdf0d644e499eaa2f755fab9abb77 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:34.964597 27228 raft_consensus.cc:729] T c32bdf0d644e499eaa2f755fab9abb77 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 868e6e4358f4475bb45967b05eccb09f, State: Initialized, Role: FOLLOWER
I20240502 01:52:34.964699 27225 consensus_queue.cc:260] T 895cfddfe1804d34a80b8a52e00efce2 P d1e7d0eabafa46f382e19f596e2fc00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:34.965023 26556 leader_election.cc:302] T dc44aad1f4ab4469a29d280a11268be0 P d1e7d0eabafa46f382e19f596e2fc00d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1e7d0eabafa46f382e19f596e2fc00d, f215e8fdc8cb45f9a5eff7cb9e6ccfce; no voters: 868e6e4358f4475bb45967b05eccb09f
I20240502 01:52:34.965577 27233 raft_consensus.cc:2781] T dc44aad1f4ab4469a29d280a11268be0 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:34.965487 27228 consensus_queue.cc:260] T c32bdf0d644e499eaa2f755fab9abb77 P 868e6e4358f4475bb45967b05eccb09f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } }
I20240502 01:52:34.965894 27233 raft_consensus.cc:686] T dc44aad1f4ab4469a29d280a11268be0 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 LEADER]: Becoming Leader. State: Replica: d1e7d0eabafa46f382e19f596e2fc00d, State: Running, Role: LEADER
I20240502 01:52:34.966004 27225 ts_tablet_manager.cc:1398] T 895cfddfe1804d34a80b8a52e00efce2 P d1e7d0eabafa46f382e19f596e2fc00d: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:52:34.967643 27228 ts_tablet_manager.cc:1398] T c32bdf0d644e499eaa2f755fab9abb77 P 868e6e4358f4475bb45967b05eccb09f: Time spent starting tablet: real 0.014s	user 0.013s	sys 0.000s
I20240502 01:52:34.966517 27233 consensus_queue.cc:237] T dc44aad1f4ab4469a29d280a11268be0 P d1e7d0eabafa46f382e19f596e2fc00d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:34.968122 27228 tablet_bootstrap.cc:492] T dc44aad1f4ab4469a29d280a11268be0 P 868e6e4358f4475bb45967b05eccb09f: Bootstrap starting.
I20240502 01:52:34.969868 27228 tablet_bootstrap.cc:654] T dc44aad1f4ab4469a29d280a11268be0 P 868e6e4358f4475bb45967b05eccb09f: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:34.972038 27228 tablet_bootstrap.cc:492] T dc44aad1f4ab4469a29d280a11268be0 P 868e6e4358f4475bb45967b05eccb09f: No bootstrap required, opened a new log
I20240502 01:52:34.972203 27228 ts_tablet_manager.cc:1367] T dc44aad1f4ab4469a29d280a11268be0 P 868e6e4358f4475bb45967b05eccb09f: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240502 01:52:34.973367 27228 raft_consensus.cc:348] T dc44aad1f4ab4469a29d280a11268be0 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:34.973541 27228 raft_consensus.cc:374] T dc44aad1f4ab4469a29d280a11268be0 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:34.973604 27228 raft_consensus.cc:729] T dc44aad1f4ab4469a29d280a11268be0 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 868e6e4358f4475bb45967b05eccb09f, State: Initialized, Role: FOLLOWER
I20240502 01:52:34.973829 27228 consensus_queue.cc:260] T dc44aad1f4ab4469a29d280a11268be0 P 868e6e4358f4475bb45967b05eccb09f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:34.973618 26081 catalog_manager.cc:5484] T dc44aad1f4ab4469a29d280a11268be0 P d1e7d0eabafa46f382e19f596e2fc00d reported cstate change: term changed from 0 to 1, leader changed from <none> to d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132). New cstate: current_term: 1 leader_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:34.974313 27228 ts_tablet_manager.cc:1398] T dc44aad1f4ab4469a29d280a11268be0 P 868e6e4358f4475bb45967b05eccb09f: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:52:34.974602 27228 tablet_bootstrap.cc:492] T 895cfddfe1804d34a80b8a52e00efce2 P 868e6e4358f4475bb45967b05eccb09f: Bootstrap starting.
I20240502 01:52:34.976181 27228 tablet_bootstrap.cc:654] T 895cfddfe1804d34a80b8a52e00efce2 P 868e6e4358f4475bb45967b05eccb09f: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:34.978277 27228 tablet_bootstrap.cc:492] T 895cfddfe1804d34a80b8a52e00efce2 P 868e6e4358f4475bb45967b05eccb09f: No bootstrap required, opened a new log
I20240502 01:52:34.978451 27228 ts_tablet_manager.cc:1367] T 895cfddfe1804d34a80b8a52e00efce2 P 868e6e4358f4475bb45967b05eccb09f: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240502 01:52:34.979064 27228 raft_consensus.cc:348] T 895cfddfe1804d34a80b8a52e00efce2 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:34.979363 27228 raft_consensus.cc:374] T 895cfddfe1804d34a80b8a52e00efce2 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:34.979467 27228 raft_consensus.cc:729] T 895cfddfe1804d34a80b8a52e00efce2 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 868e6e4358f4475bb45967b05eccb09f, State: Initialized, Role: FOLLOWER
I20240502 01:52:34.979727 27228 consensus_queue.cc:260] T 895cfddfe1804d34a80b8a52e00efce2 P 868e6e4358f4475bb45967b05eccb09f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:34.980291 27228 ts_tablet_manager.cc:1398] T 895cfddfe1804d34a80b8a52e00efce2 P 868e6e4358f4475bb45967b05eccb09f: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
W20240502 01:52:34.983970 26669 tablet.cc:2369] T 895cfddfe1804d34a80b8a52e00efce2 P d1e7d0eabafa46f382e19f596e2fc00d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:52:35.038769 26536 tablet.cc:2369] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:52:35.071993 26403 tablet.cc:2369] T 895cfddfe1804d34a80b8a52e00efce2 P 868e6e4358f4475bb45967b05eccb09f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:52:35.084637 27232 raft_consensus.cc:482] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:35.084925 27232 raft_consensus.cc:504] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:35.086761 27232 leader_election.cc:288] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132:37695), 868e6e4358f4475bb45967b05eccb09f (127.25.109.130:45571)
I20240502 01:52:35.094368 26622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "895cfddfe1804d34a80b8a52e00efce2" candidate_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" is_pre_election: true
I20240502 01:52:35.094732 26356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "895cfddfe1804d34a80b8a52e00efce2" candidate_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "868e6e4358f4475bb45967b05eccb09f" is_pre_election: true
I20240502 01:52:35.094847 26622 raft_consensus.cc:2454] T 895cfddfe1804d34a80b8a52e00efce2 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c70fbdc8a0b42ad8c09c6dbc842cc96 in term 0.
I20240502 01:52:35.095106 26356 raft_consensus.cc:2454] T 895cfddfe1804d34a80b8a52e00efce2 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c70fbdc8a0b42ad8c09c6dbc842cc96 in term 0.
I20240502 01:52:35.095501 26423 leader_election.cc:302] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c70fbdc8a0b42ad8c09c6dbc842cc96, d1e7d0eabafa46f382e19f596e2fc00d; no voters: 
I20240502 01:52:35.095945 27232 raft_consensus.cc:2781] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:35.096092 27232 raft_consensus.cc:482] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:35.096192 27232 raft_consensus.cc:3037] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:35.098263 27232 raft_consensus.cc:504] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:35.098904 27232 leader_election.cc:288] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [CANDIDATE]: Term 1 election: Requested vote from peers d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132:37695), 868e6e4358f4475bb45967b05eccb09f (127.25.109.130:45571)
I20240502 01:52:35.099407 26622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "895cfddfe1804d34a80b8a52e00efce2" candidate_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e7d0eabafa46f382e19f596e2fc00d"
I20240502 01:52:35.099481 26356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "895cfddfe1804d34a80b8a52e00efce2" candidate_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "868e6e4358f4475bb45967b05eccb09f"
I20240502 01:52:35.099591 26622 raft_consensus.cc:3037] T 895cfddfe1804d34a80b8a52e00efce2 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:35.099710 26356 raft_consensus.cc:3037] T 895cfddfe1804d34a80b8a52e00efce2 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:35.101663 26622 raft_consensus.cc:2454] T 895cfddfe1804d34a80b8a52e00efce2 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c70fbdc8a0b42ad8c09c6dbc842cc96 in term 1.
I20240502 01:52:35.101934 26356 raft_consensus.cc:2454] T 895cfddfe1804d34a80b8a52e00efce2 P 868e6e4358f4475bb45967b05eccb09f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c70fbdc8a0b42ad8c09c6dbc842cc96 in term 1.
I20240502 01:52:35.102133 26423 leader_election.cc:302] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c70fbdc8a0b42ad8c09c6dbc842cc96, d1e7d0eabafa46f382e19f596e2fc00d; no voters: 
I20240502 01:52:35.102488 27232 raft_consensus.cc:2781] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:35.102948 27232 raft_consensus.cc:686] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 LEADER]: Becoming Leader. State: Replica: 6c70fbdc8a0b42ad8c09c6dbc842cc96, State: Running, Role: LEADER
I20240502 01:52:35.103412 27232 consensus_queue.cc:237] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
W20240502 01:52:35.108215 26271 tablet.cc:2369] T c32bdf0d644e499eaa2f755fab9abb77 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:52:35.108171 26080 catalog_manager.cc:5484] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131). New cstate: current_term: 1 leader_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:35.127907 27242 raft_consensus.cc:482] T c32bdf0d644e499eaa2f755fab9abb77 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:35.128129 27242 raft_consensus.cc:504] T c32bdf0d644e499eaa2f755fab9abb77 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } }
I20240502 01:52:35.130360 27242 leader_election.cc:288] T c32bdf0d644e499eaa2f755fab9abb77 P 868e6e4358f4475bb45967b05eccb09f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541), 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131:38693)
I20240502 01:52:35.137223 26224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32bdf0d644e499eaa2f755fab9abb77" candidate_uuid: "868e6e4358f4475bb45967b05eccb09f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" is_pre_election: true
I20240502 01:52:35.137537 26224 raft_consensus.cc:2454] T c32bdf0d644e499eaa2f755fab9abb77 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 868e6e4358f4475bb45967b05eccb09f in term 0.
I20240502 01:52:35.138015 26292 leader_election.cc:302] T c32bdf0d644e499eaa2f755fab9abb77 P 868e6e4358f4475bb45967b05eccb09f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 868e6e4358f4475bb45967b05eccb09f, f215e8fdc8cb45f9a5eff7cb9e6ccfce; no voters: 
I20240502 01:52:35.138186 26490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32bdf0d644e499eaa2f755fab9abb77" candidate_uuid: "868e6e4358f4475bb45967b05eccb09f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" is_pre_election: true
I20240502 01:52:35.138415 27242 raft_consensus.cc:2781] T c32bdf0d644e499eaa2f755fab9abb77 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:35.138536 27242 raft_consensus.cc:482] T c32bdf0d644e499eaa2f755fab9abb77 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:35.138630 27242 raft_consensus.cc:3037] T c32bdf0d644e499eaa2f755fab9abb77 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:35.138681 26490 raft_consensus.cc:2454] T c32bdf0d644e499eaa2f755fab9abb77 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 868e6e4358f4475bb45967b05eccb09f in term 0.
I20240502 01:52:35.140928 27242 raft_consensus.cc:504] T c32bdf0d644e499eaa2f755fab9abb77 P 868e6e4358f4475bb45967b05eccb09f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } }
I20240502 01:52:35.141553 27242 leader_election.cc:288] T c32bdf0d644e499eaa2f755fab9abb77 P 868e6e4358f4475bb45967b05eccb09f [CANDIDATE]: Term 1 election: Requested vote from peers f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541), 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131:38693)
I20240502 01:52:35.142107 26224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32bdf0d644e499eaa2f755fab9abb77" candidate_uuid: "868e6e4358f4475bb45967b05eccb09f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce"
I20240502 01:52:35.142170 26490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32bdf0d644e499eaa2f755fab9abb77" candidate_uuid: "868e6e4358f4475bb45967b05eccb09f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96"
I20240502 01:52:35.142364 26490 raft_consensus.cc:3037] T c32bdf0d644e499eaa2f755fab9abb77 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:35.142385 26224 raft_consensus.cc:3037] T c32bdf0d644e499eaa2f755fab9abb77 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:35.145334 26490 raft_consensus.cc:2454] T c32bdf0d644e499eaa2f755fab9abb77 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 868e6e4358f4475bb45967b05eccb09f in term 1.
I20240502 01:52:35.145323 26224 raft_consensus.cc:2454] T c32bdf0d644e499eaa2f755fab9abb77 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 868e6e4358f4475bb45967b05eccb09f in term 1.
I20240502 01:52:35.145838 26292 leader_election.cc:302] T c32bdf0d644e499eaa2f755fab9abb77 P 868e6e4358f4475bb45967b05eccb09f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c70fbdc8a0b42ad8c09c6dbc842cc96, 868e6e4358f4475bb45967b05eccb09f; no voters: 
I20240502 01:52:35.146199 27242 raft_consensus.cc:2781] T c32bdf0d644e499eaa2f755fab9abb77 P 868e6e4358f4475bb45967b05eccb09f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:35.146677 27242 raft_consensus.cc:686] T c32bdf0d644e499eaa2f755fab9abb77 P 868e6e4358f4475bb45967b05eccb09f [term 1 LEADER]: Becoming Leader. State: Replica: 868e6e4358f4475bb45967b05eccb09f, State: Running, Role: LEADER
I20240502 01:52:35.147163 27242 consensus_queue.cc:237] T c32bdf0d644e499eaa2f755fab9abb77 P 868e6e4358f4475bb45967b05eccb09f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } }
I20240502 01:52:35.152290 26080 catalog_manager.cc:5484] T c32bdf0d644e499eaa2f755fab9abb77 P 868e6e4358f4475bb45967b05eccb09f reported cstate change: term changed from 0 to 1, leader changed from <none> to 868e6e4358f4475bb45967b05eccb09f (127.25.109.130). New cstate: current_term: 1 leader_uuid: "868e6e4358f4475bb45967b05eccb09f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:35.434319 26356 raft_consensus.cc:3037] T dc44aad1f4ab4469a29d280a11268be0 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:35.437597 27233 consensus_queue.cc:1035] T dc44aad1f4ab4469a29d280a11268be0 P d1e7d0eabafa46f382e19f596e2fc00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:35.445433 27236 consensus_queue.cc:1035] T dc44aad1f4ab4469a29d280a11268be0 P d1e7d0eabafa46f382e19f596e2fc00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:35.541077 27242 consensus_queue.cc:1035] T c32bdf0d644e499eaa2f755fab9abb77 P 868e6e4358f4475bb45967b05eccb09f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:35.546160 27252 consensus_queue.cc:1035] T c32bdf0d644e499eaa2f755fab9abb77 P 868e6e4358f4475bb45967b05eccb09f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:35.548722 27232 consensus_queue.cc:1035] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:35.564502 27248 consensus_queue.cc:1035] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:52:35.575275 27254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:35.575620 27254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:35.583276 27254 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'

INSERT report
    rows total: 2000
    time total: 224.81 ms
  time per row: 0.112405 ms
I20240502 01:52:35.960705 26081 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57636:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:52:35.969914 26204 tablet_service.cc:1467] Processing CreateTablet for tablet e425e061e59a4bffa56b498fdf462a58 (DEFAULT_TABLE table=rebalance_test_table_1 [id=33f02a7f80f145fd914290afb2863661]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:35.970115 26468 tablet_service.cc:1467] Processing CreateTablet for tablet e425e061e59a4bffa56b498fdf462a58 (DEFAULT_TABLE table=rebalance_test_table_1 [id=33f02a7f80f145fd914290afb2863661]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:35.970470 26204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e425e061e59a4bffa56b498fdf462a58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:35.970638 26468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e425e061e59a4bffa56b498fdf462a58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:35.971092 26469 tablet_service.cc:1467] Processing CreateTablet for tablet 4e6854ffbc944e45b51256143505cdb8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=33f02a7f80f145fd914290afb2863661]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:35.971187 26335 tablet_service.cc:1467] Processing CreateTablet for tablet 7def725255ba4448837b47ba65a6ed44 (DEFAULT_TABLE table=rebalance_test_table_1 [id=33f02a7f80f145fd914290afb2863661]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:35.971452 26469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e6854ffbc944e45b51256143505cdb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:35.971573 26335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7def725255ba4448837b47ba65a6ed44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:35.976981 26600 tablet_service.cc:1467] Processing CreateTablet for tablet e425e061e59a4bffa56b498fdf462a58 (DEFAULT_TABLE table=rebalance_test_table_1 [id=33f02a7f80f145fd914290afb2863661]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:35.977463 26600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e425e061e59a4bffa56b498fdf462a58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:35.980307 26601 tablet_service.cc:1467] Processing CreateTablet for tablet 4e6854ffbc944e45b51256143505cdb8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=33f02a7f80f145fd914290afb2863661]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:35.980741 26601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e6854ffbc944e45b51256143505cdb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:35.981401 27316 tablet_bootstrap.cc:492] T 7def725255ba4448837b47ba65a6ed44 P 868e6e4358f4475bb45967b05eccb09f: Bootstrap starting.
I20240502 01:52:35.982254 27315 tablet_bootstrap.cc:492] T e425e061e59a4bffa56b498fdf462a58 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Bootstrap starting.
I20240502 01:52:35.982712 26467 tablet_service.cc:1467] Processing CreateTablet for tablet 7def725255ba4448837b47ba65a6ed44 (DEFAULT_TABLE table=rebalance_test_table_1 [id=33f02a7f80f145fd914290afb2863661]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:35.983152 27316 tablet_bootstrap.cc:654] T 7def725255ba4448837b47ba65a6ed44 P 868e6e4358f4475bb45967b05eccb09f: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:35.983192 26467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7def725255ba4448837b47ba65a6ed44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:35.984246 27315 tablet_bootstrap.cc:654] T e425e061e59a4bffa56b498fdf462a58 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:35.986682 27315 tablet_bootstrap.cc:492] T e425e061e59a4bffa56b498fdf462a58 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: No bootstrap required, opened a new log
I20240502 01:52:35.986666 26203 tablet_service.cc:1467] Processing CreateTablet for tablet 4e6854ffbc944e45b51256143505cdb8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=33f02a7f80f145fd914290afb2863661]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:35.986781 26202 tablet_service.cc:1467] Processing CreateTablet for tablet 7def725255ba4448837b47ba65a6ed44 (DEFAULT_TABLE table=rebalance_test_table_1 [id=33f02a7f80f145fd914290afb2863661]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:35.986944 27315 ts_tablet_manager.cc:1367] T e425e061e59a4bffa56b498fdf462a58 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240502 01:52:35.987152 26203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e6854ffbc944e45b51256143505cdb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:35.989626 27315 raft_consensus.cc:348] T e425e061e59a4bffa56b498fdf462a58 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:52:35.989953 27315 raft_consensus.cc:374] T e425e061e59a4bffa56b498fdf462a58 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:35.990074 27315 raft_consensus.cc:729] T e425e061e59a4bffa56b498fdf462a58 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c70fbdc8a0b42ad8c09c6dbc842cc96, State: Initialized, Role: FOLLOWER
I20240502 01:52:35.990418 27315 consensus_queue.cc:260] T e425e061e59a4bffa56b498fdf462a58 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:52:35.991108 27315 ts_tablet_manager.cc:1398] T e425e061e59a4bffa56b498fdf462a58 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 01:52:35.991576 27315 tablet_bootstrap.cc:492] T 4e6854ffbc944e45b51256143505cdb8 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Bootstrap starting.
I20240502 01:52:35.993047 26202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7def725255ba4448837b47ba65a6ed44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:35.994616 27315 tablet_bootstrap.cc:654] T 4e6854ffbc944e45b51256143505cdb8 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:36.013890 27320 tablet_bootstrap.cc:492] T 7def725255ba4448837b47ba65a6ed44 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap starting.
I20240502 01:52:36.014568 27321 tablet_bootstrap.cc:492] T 4e6854ffbc944e45b51256143505cdb8 P d1e7d0eabafa46f382e19f596e2fc00d: Bootstrap starting.
I20240502 01:52:36.015677 27315 tablet_bootstrap.cc:492] T 4e6854ffbc944e45b51256143505cdb8 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: No bootstrap required, opened a new log
I20240502 01:52:36.016104 27315 ts_tablet_manager.cc:1367] T 4e6854ffbc944e45b51256143505cdb8 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Time spent bootstrapping tablet: real 0.025s	user 0.006s	sys 0.017s
I20240502 01:52:36.016889 27321 tablet_bootstrap.cc:654] T 4e6854ffbc944e45b51256143505cdb8 P d1e7d0eabafa46f382e19f596e2fc00d: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:36.016889 27320 tablet_bootstrap.cc:654] T 7def725255ba4448837b47ba65a6ed44 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:36.017333 27315 raft_consensus.cc:348] T 4e6854ffbc944e45b51256143505cdb8 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:52:36.017668 27315 raft_consensus.cc:374] T 4e6854ffbc944e45b51256143505cdb8 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:36.017781 27315 raft_consensus.cc:729] T 4e6854ffbc944e45b51256143505cdb8 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c70fbdc8a0b42ad8c09c6dbc842cc96, State: Initialized, Role: FOLLOWER
I20240502 01:52:36.018196 27315 consensus_queue.cc:260] T 4e6854ffbc944e45b51256143505cdb8 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:52:36.019472 27315 ts_tablet_manager.cc:1398] T 4e6854ffbc944e45b51256143505cdb8 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:52:36.019927 27315 tablet_bootstrap.cc:492] T 7def725255ba4448837b47ba65a6ed44 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Bootstrap starting.
I20240502 01:52:36.021699 27315 tablet_bootstrap.cc:654] T 7def725255ba4448837b47ba65a6ed44 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:36.022195 27316 tablet_bootstrap.cc:492] T 7def725255ba4448837b47ba65a6ed44 P 868e6e4358f4475bb45967b05eccb09f: No bootstrap required, opened a new log
I20240502 01:52:36.022379 27316 ts_tablet_manager.cc:1367] T 7def725255ba4448837b47ba65a6ed44 P 868e6e4358f4475bb45967b05eccb09f: Time spent bootstrapping tablet: real 0.041s	user 0.008s	sys 0.032s
I20240502 01:52:36.023308 27316 raft_consensus.cc:348] T 7def725255ba4448837b47ba65a6ed44 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } }
I20240502 01:52:36.023551 27316 raft_consensus.cc:374] T 7def725255ba4448837b47ba65a6ed44 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:36.023643 27316 raft_consensus.cc:729] T 7def725255ba4448837b47ba65a6ed44 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 868e6e4358f4475bb45967b05eccb09f, State: Initialized, Role: FOLLOWER
I20240502 01:52:36.023979 27316 consensus_queue.cc:260] T 7def725255ba4448837b47ba65a6ed44 P 868e6e4358f4475bb45967b05eccb09f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } }
I20240502 01:52:36.024780 27316 ts_tablet_manager.cc:1398] T 7def725255ba4448837b47ba65a6ed44 P 868e6e4358f4475bb45967b05eccb09f: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.003s
I20240502 01:52:36.026077 27320 tablet_bootstrap.cc:492] T 7def725255ba4448837b47ba65a6ed44 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: No bootstrap required, opened a new log
I20240502 01:52:36.026258 27320 ts_tablet_manager.cc:1367] T 7def725255ba4448837b47ba65a6ed44 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent bootstrapping tablet: real 0.013s	user 0.003s	sys 0.003s
I20240502 01:52:36.027179 27320 raft_consensus.cc:348] T 7def725255ba4448837b47ba65a6ed44 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } }
I20240502 01:52:36.027431 27320 raft_consensus.cc:374] T 7def725255ba4448837b47ba65a6ed44 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:36.027520 27320 raft_consensus.cc:729] T 7def725255ba4448837b47ba65a6ed44 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f215e8fdc8cb45f9a5eff7cb9e6ccfce, State: Initialized, Role: FOLLOWER
I20240502 01:52:36.027917 27320 consensus_queue.cc:260] T 7def725255ba4448837b47ba65a6ed44 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } }
I20240502 01:52:36.029436 27321 tablet_bootstrap.cc:492] T 4e6854ffbc944e45b51256143505cdb8 P d1e7d0eabafa46f382e19f596e2fc00d: No bootstrap required, opened a new log
I20240502 01:52:36.029613 27321 ts_tablet_manager.cc:1367] T 4e6854ffbc944e45b51256143505cdb8 P d1e7d0eabafa46f382e19f596e2fc00d: Time spent bootstrapping tablet: real 0.015s	user 0.003s	sys 0.007s
I20240502 01:52:36.029959 27315 tablet_bootstrap.cc:492] T 7def725255ba4448837b47ba65a6ed44 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: No bootstrap required, opened a new log
I20240502 01:52:36.030120 27315 ts_tablet_manager.cc:1367] T 7def725255ba4448837b47ba65a6ed44 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Time spent bootstrapping tablet: real 0.010s	user 0.002s	sys 0.006s
I20240502 01:52:36.030175 27320 ts_tablet_manager.cc:1398] T 7def725255ba4448837b47ba65a6ed44 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent starting tablet: real 0.004s	user 0.003s	sys 0.000s
I20240502 01:52:36.030501 27320 tablet_bootstrap.cc:492] T 4e6854ffbc944e45b51256143505cdb8 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap starting.
I20240502 01:52:36.030480 27321 raft_consensus.cc:348] T 4e6854ffbc944e45b51256143505cdb8 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:52:36.030686 27321 raft_consensus.cc:374] T 4e6854ffbc944e45b51256143505cdb8 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:36.030771 27321 raft_consensus.cc:729] T 4e6854ffbc944e45b51256143505cdb8 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e7d0eabafa46f382e19f596e2fc00d, State: Initialized, Role: FOLLOWER
I20240502 01:52:36.030987 27315 raft_consensus.cc:348] T 7def725255ba4448837b47ba65a6ed44 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } }
I20240502 01:52:36.031033 27321 consensus_queue.cc:260] T 4e6854ffbc944e45b51256143505cdb8 P d1e7d0eabafa46f382e19f596e2fc00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:52:36.031200 27315 raft_consensus.cc:374] T 7def725255ba4448837b47ba65a6ed44 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:36.031293 27315 raft_consensus.cc:729] T 7def725255ba4448837b47ba65a6ed44 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c70fbdc8a0b42ad8c09c6dbc842cc96, State: Initialized, Role: FOLLOWER
I20240502 01:52:36.031526 27315 consensus_queue.cc:260] T 7def725255ba4448837b47ba65a6ed44 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } }
I20240502 01:52:36.032344 27320 tablet_bootstrap.cc:654] T 4e6854ffbc944e45b51256143505cdb8 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:36.033941 27321 ts_tablet_manager.cc:1398] T 4e6854ffbc944e45b51256143505cdb8 P d1e7d0eabafa46f382e19f596e2fc00d: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.001s
I20240502 01:52:36.034404 27315 ts_tablet_manager.cc:1398] T 7def725255ba4448837b47ba65a6ed44 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.001s
I20240502 01:52:36.034915 27321 tablet_bootstrap.cc:492] T e425e061e59a4bffa56b498fdf462a58 P d1e7d0eabafa46f382e19f596e2fc00d: Bootstrap starting.
I20240502 01:52:36.036567 27321 tablet_bootstrap.cc:654] T e425e061e59a4bffa56b498fdf462a58 P d1e7d0eabafa46f382e19f596e2fc00d: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:36.038357 27320 tablet_bootstrap.cc:492] T 4e6854ffbc944e45b51256143505cdb8 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: No bootstrap required, opened a new log
I20240502 01:52:36.038550 27320 ts_tablet_manager.cc:1367] T 4e6854ffbc944e45b51256143505cdb8 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent bootstrapping tablet: real 0.008s	user 0.006s	sys 0.000s
I20240502 01:52:36.038983 27321 tablet_bootstrap.cc:492] T e425e061e59a4bffa56b498fdf462a58 P d1e7d0eabafa46f382e19f596e2fc00d: No bootstrap required, opened a new log
I20240502 01:52:36.039136 27321 ts_tablet_manager.cc:1367] T e425e061e59a4bffa56b498fdf462a58 P d1e7d0eabafa46f382e19f596e2fc00d: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240502 01:52:36.039208 27320 raft_consensus.cc:348] T 4e6854ffbc944e45b51256143505cdb8 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:52:36.039417 27320 raft_consensus.cc:374] T 4e6854ffbc944e45b51256143505cdb8 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:36.039492 27320 raft_consensus.cc:729] T 4e6854ffbc944e45b51256143505cdb8 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f215e8fdc8cb45f9a5eff7cb9e6ccfce, State: Initialized, Role: FOLLOWER
I20240502 01:52:36.039731 27320 consensus_queue.cc:260] T 4e6854ffbc944e45b51256143505cdb8 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:52:36.039794 27321 raft_consensus.cc:348] T e425e061e59a4bffa56b498fdf462a58 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:52:36.040045 27321 raft_consensus.cc:374] T e425e061e59a4bffa56b498fdf462a58 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:36.040134 27321 raft_consensus.cc:729] T e425e061e59a4bffa56b498fdf462a58 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e7d0eabafa46f382e19f596e2fc00d, State: Initialized, Role: FOLLOWER
I20240502 01:52:36.040257 27320 ts_tablet_manager.cc:1398] T 4e6854ffbc944e45b51256143505cdb8 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:52:36.040378 27321 consensus_queue.cc:260] T e425e061e59a4bffa56b498fdf462a58 P d1e7d0eabafa46f382e19f596e2fc00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:52:36.040613 27320 tablet_bootstrap.cc:492] T e425e061e59a4bffa56b498fdf462a58 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap starting.
I20240502 01:52:36.040930 27321 ts_tablet_manager.cc:1398] T e425e061e59a4bffa56b498fdf462a58 P d1e7d0eabafa46f382e19f596e2fc00d: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 01:52:36.042217 27320 tablet_bootstrap.cc:654] T e425e061e59a4bffa56b498fdf462a58 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:36.044368 27320 tablet_bootstrap.cc:492] T e425e061e59a4bffa56b498fdf462a58 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: No bootstrap required, opened a new log
I20240502 01:52:36.044529 27320 ts_tablet_manager.cc:1367] T e425e061e59a4bffa56b498fdf462a58 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240502 01:52:36.045307 27320 raft_consensus.cc:348] T e425e061e59a4bffa56b498fdf462a58 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:52:36.045485 27320 raft_consensus.cc:374] T e425e061e59a4bffa56b498fdf462a58 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:36.045578 27320 raft_consensus.cc:729] T e425e061e59a4bffa56b498fdf462a58 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f215e8fdc8cb45f9a5eff7cb9e6ccfce, State: Initialized, Role: FOLLOWER
I20240502 01:52:36.045776 27320 consensus_queue.cc:260] T e425e061e59a4bffa56b498fdf462a58 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:52:36.046257 27320 ts_tablet_manager.cc:1398] T e425e061e59a4bffa56b498fdf462a58 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:52:36.048125 27259 raft_consensus.cc:482] T e425e061e59a4bffa56b498fdf462a58 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:36.048285 27259 raft_consensus.cc:504] T e425e061e59a4bffa56b498fdf462a58 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:52:36.050128 27259 leader_election.cc:288] T e425e061e59a4bffa56b498fdf462a58 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131:38693), d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132:37695)
I20240502 01:52:36.057085 26622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e425e061e59a4bffa56b498fdf462a58" candidate_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" is_pre_election: true
I20240502 01:52:36.057085 26490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e425e061e59a4bffa56b498fdf462a58" candidate_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" is_pre_election: true
I20240502 01:52:36.057400 26622 raft_consensus.cc:2454] T e425e061e59a4bffa56b498fdf462a58 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f215e8fdc8cb45f9a5eff7cb9e6ccfce in term 0.
I20240502 01:52:36.057400 26490 raft_consensus.cc:2454] T e425e061e59a4bffa56b498fdf462a58 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f215e8fdc8cb45f9a5eff7cb9e6ccfce in term 0.
I20240502 01:52:36.057929 26158 leader_election.cc:302] T e425e061e59a4bffa56b498fdf462a58 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c70fbdc8a0b42ad8c09c6dbc842cc96, f215e8fdc8cb45f9a5eff7cb9e6ccfce; no voters: 
I20240502 01:52:36.058341 27259 raft_consensus.cc:2781] T e425e061e59a4bffa56b498fdf462a58 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:36.058475 27259 raft_consensus.cc:482] T e425e061e59a4bffa56b498fdf462a58 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:36.058579 27259 raft_consensus.cc:3037] T e425e061e59a4bffa56b498fdf462a58 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:36.060729 27259 raft_consensus.cc:504] T e425e061e59a4bffa56b498fdf462a58 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:52:36.061553 27259 leader_election.cc:288] T e425e061e59a4bffa56b498fdf462a58 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [CANDIDATE]: Term 1 election: Requested vote from peers 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131:38693), d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132:37695)
I20240502 01:52:36.062085 26490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e425e061e59a4bffa56b498fdf462a58" candidate_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96"
I20240502 01:52:36.062105 26622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e425e061e59a4bffa56b498fdf462a58" candidate_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e7d0eabafa46f382e19f596e2fc00d"
I20240502 01:52:36.062307 26490 raft_consensus.cc:3037] T e425e061e59a4bffa56b498fdf462a58 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:36.062355 26622 raft_consensus.cc:3037] T e425e061e59a4bffa56b498fdf462a58 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:36.064278 26490 raft_consensus.cc:2454] T e425e061e59a4bffa56b498fdf462a58 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f215e8fdc8cb45f9a5eff7cb9e6ccfce in term 1.
I20240502 01:52:36.064294 26622 raft_consensus.cc:2454] T e425e061e59a4bffa56b498fdf462a58 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f215e8fdc8cb45f9a5eff7cb9e6ccfce in term 1.
I20240502 01:52:36.064772 26158 leader_election.cc:302] T e425e061e59a4bffa56b498fdf462a58 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6c70fbdc8a0b42ad8c09c6dbc842cc96, d1e7d0eabafa46f382e19f596e2fc00d, f215e8fdc8cb45f9a5eff7cb9e6ccfce; no voters: 
I20240502 01:52:36.065109 27259 raft_consensus.cc:2781] T e425e061e59a4bffa56b498fdf462a58 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:36.065562 27259 raft_consensus.cc:686] T e425e061e59a4bffa56b498fdf462a58 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 LEADER]: Becoming Leader. State: Replica: f215e8fdc8cb45f9a5eff7cb9e6ccfce, State: Running, Role: LEADER
I20240502 01:52:36.066012 27259 consensus_queue.cc:237] T e425e061e59a4bffa56b498fdf462a58 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:52:36.070138 26077 catalog_manager.cc:5484] T e425e061e59a4bffa56b498fdf462a58 P f215e8fdc8cb45f9a5eff7cb9e6ccfce reported cstate change: term changed from 0 to 1, leader changed from <none> to f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129). New cstate: current_term: 1 leader_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:36.123529 27252 raft_consensus.cc:482] T 7def725255ba4448837b47ba65a6ed44 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:36.123766 27252 raft_consensus.cc:504] T 7def725255ba4448837b47ba65a6ed44 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } }
I20240502 01:52:36.124502 27252 leader_election.cc:288] T 7def725255ba4448837b47ba65a6ed44 P 868e6e4358f4475bb45967b05eccb09f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541), 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131:38693)
I20240502 01:52:36.125144 26223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7def725255ba4448837b47ba65a6ed44" candidate_uuid: "868e6e4358f4475bb45967b05eccb09f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" is_pre_election: true
I20240502 01:52:36.125192 26490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7def725255ba4448837b47ba65a6ed44" candidate_uuid: "868e6e4358f4475bb45967b05eccb09f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" is_pre_election: true
I20240502 01:52:36.125375 26223 raft_consensus.cc:2454] T 7def725255ba4448837b47ba65a6ed44 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 868e6e4358f4475bb45967b05eccb09f in term 0.
I20240502 01:52:36.125445 26490 raft_consensus.cc:2454] T 7def725255ba4448837b47ba65a6ed44 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 868e6e4358f4475bb45967b05eccb09f in term 0.
I20240502 01:52:36.125851 26292 leader_election.cc:302] T 7def725255ba4448837b47ba65a6ed44 P 868e6e4358f4475bb45967b05eccb09f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 868e6e4358f4475bb45967b05eccb09f, f215e8fdc8cb45f9a5eff7cb9e6ccfce; no voters: 
I20240502 01:52:36.126149 27252 raft_consensus.cc:2781] T 7def725255ba4448837b47ba65a6ed44 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:36.126245 27252 raft_consensus.cc:482] T 7def725255ba4448837b47ba65a6ed44 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:36.126350 27252 raft_consensus.cc:3037] T 7def725255ba4448837b47ba65a6ed44 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:36.128281 27252 raft_consensus.cc:504] T 7def725255ba4448837b47ba65a6ed44 P 868e6e4358f4475bb45967b05eccb09f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } }
I20240502 01:52:36.128861 27252 leader_election.cc:288] T 7def725255ba4448837b47ba65a6ed44 P 868e6e4358f4475bb45967b05eccb09f [CANDIDATE]: Term 1 election: Requested vote from peers f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541), 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131:38693)
I20240502 01:52:36.129348 26223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7def725255ba4448837b47ba65a6ed44" candidate_uuid: "868e6e4358f4475bb45967b05eccb09f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce"
I20240502 01:52:36.129436 26490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7def725255ba4448837b47ba65a6ed44" candidate_uuid: "868e6e4358f4475bb45967b05eccb09f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96"
I20240502 01:52:36.129528 26223 raft_consensus.cc:3037] T 7def725255ba4448837b47ba65a6ed44 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:36.129643 26490 raft_consensus.cc:3037] T 7def725255ba4448837b47ba65a6ed44 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:36.131423 26223 raft_consensus.cc:2454] T 7def725255ba4448837b47ba65a6ed44 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 868e6e4358f4475bb45967b05eccb09f in term 1.
I20240502 01:52:36.131507 26490 raft_consensus.cc:2454] T 7def725255ba4448837b47ba65a6ed44 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 868e6e4358f4475bb45967b05eccb09f in term 1.
I20240502 01:52:36.131955 26292 leader_election.cc:302] T 7def725255ba4448837b47ba65a6ed44 P 868e6e4358f4475bb45967b05eccb09f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 868e6e4358f4475bb45967b05eccb09f, f215e8fdc8cb45f9a5eff7cb9e6ccfce; no voters: 
I20240502 01:52:36.132259 27252 raft_consensus.cc:2781] T 7def725255ba4448837b47ba65a6ed44 P 868e6e4358f4475bb45967b05eccb09f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:36.132426 27252 raft_consensus.cc:686] T 7def725255ba4448837b47ba65a6ed44 P 868e6e4358f4475bb45967b05eccb09f [term 1 LEADER]: Becoming Leader. State: Replica: 868e6e4358f4475bb45967b05eccb09f, State: Running, Role: LEADER
I20240502 01:52:36.132659 27252 consensus_queue.cc:237] T 7def725255ba4448837b47ba65a6ed44 P 868e6e4358f4475bb45967b05eccb09f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } }
I20240502 01:52:36.135635 26077 catalog_manager.cc:5484] T 7def725255ba4448837b47ba65a6ed44 P 868e6e4358f4475bb45967b05eccb09f reported cstate change: term changed from 0 to 1, leader changed from <none> to 868e6e4358f4475bb45967b05eccb09f (127.25.109.130). New cstate: current_term: 1 leader_uuid: "868e6e4358f4475bb45967b05eccb09f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:36.146411 27259 raft_consensus.cc:482] T 4e6854ffbc944e45b51256143505cdb8 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:36.146620 27259 raft_consensus.cc:504] T 4e6854ffbc944e45b51256143505cdb8 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:52:36.147266 27259 leader_election.cc:288] T 4e6854ffbc944e45b51256143505cdb8 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131:38693), d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132:37695)
I20240502 01:52:36.147900 26490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e6854ffbc944e45b51256143505cdb8" candidate_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" is_pre_election: true
I20240502 01:52:36.147917 26622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e6854ffbc944e45b51256143505cdb8" candidate_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" is_pre_election: true
I20240502 01:52:36.148133 26622 raft_consensus.cc:2454] T 4e6854ffbc944e45b51256143505cdb8 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f215e8fdc8cb45f9a5eff7cb9e6ccfce in term 0.
I20240502 01:52:36.148149 26490 raft_consensus.cc:2454] T 4e6854ffbc944e45b51256143505cdb8 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f215e8fdc8cb45f9a5eff7cb9e6ccfce in term 0.
I20240502 01:52:36.148620 26157 leader_election.cc:302] T 4e6854ffbc944e45b51256143505cdb8 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e7d0eabafa46f382e19f596e2fc00d, f215e8fdc8cb45f9a5eff7cb9e6ccfce; no voters: 
I20240502 01:52:36.148905 27259 raft_consensus.cc:2781] T 4e6854ffbc944e45b51256143505cdb8 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:36.148995 27259 raft_consensus.cc:482] T 4e6854ffbc944e45b51256143505cdb8 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:36.149096 27259 raft_consensus.cc:3037] T 4e6854ffbc944e45b51256143505cdb8 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:36.151156 27259 raft_consensus.cc:504] T 4e6854ffbc944e45b51256143505cdb8 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:52:36.151850 27259 leader_election.cc:288] T 4e6854ffbc944e45b51256143505cdb8 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [CANDIDATE]: Term 1 election: Requested vote from peers 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131:38693), d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132:37695)
I20240502 01:52:36.152549 26622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e6854ffbc944e45b51256143505cdb8" candidate_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e7d0eabafa46f382e19f596e2fc00d"
I20240502 01:52:36.152565 26490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e6854ffbc944e45b51256143505cdb8" candidate_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96"
I20240502 01:52:36.152763 26622 raft_consensus.cc:3037] T 4e6854ffbc944e45b51256143505cdb8 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:36.152798 26490 raft_consensus.cc:3037] T 4e6854ffbc944e45b51256143505cdb8 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:36.154789 26490 raft_consensus.cc:2454] T 4e6854ffbc944e45b51256143505cdb8 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f215e8fdc8cb45f9a5eff7cb9e6ccfce in term 1.
I20240502 01:52:36.154803 26622 raft_consensus.cc:2454] T 4e6854ffbc944e45b51256143505cdb8 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f215e8fdc8cb45f9a5eff7cb9e6ccfce in term 1.
I20240502 01:52:36.155329 26157 leader_election.cc:302] T 4e6854ffbc944e45b51256143505cdb8 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e7d0eabafa46f382e19f596e2fc00d, f215e8fdc8cb45f9a5eff7cb9e6ccfce; no voters: 
I20240502 01:52:36.155664 27259 raft_consensus.cc:2781] T 4e6854ffbc944e45b51256143505cdb8 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:36.155925 27259 raft_consensus.cc:686] T 4e6854ffbc944e45b51256143505cdb8 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 LEADER]: Becoming Leader. State: Replica: f215e8fdc8cb45f9a5eff7cb9e6ccfce, State: Running, Role: LEADER
I20240502 01:52:36.156211 27259 consensus_queue.cc:237] T 4e6854ffbc944e45b51256143505cdb8 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:52:36.159171 26077 catalog_manager.cc:5484] T 4e6854ffbc944e45b51256143505cdb8 P f215e8fdc8cb45f9a5eff7cb9e6ccfce reported cstate change: term changed from 0 to 1, leader changed from <none> to f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129). New cstate: current_term: 1 leader_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:36.497387 27259 consensus_queue.cc:1035] T e425e061e59a4bffa56b498fdf462a58 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:36.504669 27259 consensus_queue.cc:1035] T e425e061e59a4bffa56b498fdf462a58 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:52:36.522464 27334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:36.522934 27334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:36.531167 27334 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:52:36.561867 27259 consensus_queue.cc:1035] T 4e6854ffbc944e45b51256143505cdb8 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
I20240502 01:52:36.568781 27337 consensus_queue.cc:1035] T 4e6854ffbc944e45b51256143505cdb8 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:36.642871 27252 consensus_queue.cc:1035] T 7def725255ba4448837b47ba65a6ed44 P 868e6e4358f4475bb45967b05eccb09f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:36.651659 27252 consensus_queue.cc:1035] T 7def725255ba4448837b47ba65a6ed44 P 868e6e4358f4475bb45967b05eccb09f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 267.657 ms
  time per row: 0.133828 ms
I20240502 01:52:36.957222 26077 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57636:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:52:36.967897 26601 tablet_service.cc:1467] Processing CreateTablet for tablet 01b7ac587d134a61b333489d937c0a21 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ce4b1d786b674a8fa90b2b56dcb5cd1e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:36.968082 26202 tablet_service.cc:1467] Processing CreateTablet for tablet 01b7ac587d134a61b333489d937c0a21 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ce4b1d786b674a8fa90b2b56dcb5cd1e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:36.968467 26601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01b7ac587d134a61b333489d937c0a21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:36.968526 26202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01b7ac587d134a61b333489d937c0a21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:36.969704 26467 tablet_service.cc:1467] Processing CreateTablet for tablet d4fa5ee9130340f982a6f0bcc0b28b40 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ce4b1d786b674a8fa90b2b56dcb5cd1e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:36.970044 26204 tablet_service.cc:1467] Processing CreateTablet for tablet 536039b70452486684f7075cf45c1851 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ce4b1d786b674a8fa90b2b56dcb5cd1e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:36.970211 26467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4fa5ee9130340f982a6f0bcc0b28b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:36.970501 26204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 536039b70452486684f7075cf45c1851. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:36.972918 26335 tablet_service.cc:1467] Processing CreateTablet for tablet 01b7ac587d134a61b333489d937c0a21 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ce4b1d786b674a8fa90b2b56dcb5cd1e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:36.973372 26335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01b7ac587d134a61b333489d937c0a21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:36.976471 26334 tablet_service.cc:1467] Processing CreateTablet for tablet d4fa5ee9130340f982a6f0bcc0b28b40 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ce4b1d786b674a8fa90b2b56dcb5cd1e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:36.977012 26334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4fa5ee9130340f982a6f0bcc0b28b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:36.979045 27403 tablet_bootstrap.cc:492] T 01b7ac587d134a61b333489d937c0a21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap starting.
I20240502 01:52:36.978967 26336 tablet_service.cc:1467] Processing CreateTablet for tablet 536039b70452486684f7075cf45c1851 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ce4b1d786b674a8fa90b2b56dcb5cd1e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:36.979400 26336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 536039b70452486684f7075cf45c1851. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:36.980756 26600 tablet_service.cc:1467] Processing CreateTablet for tablet d4fa5ee9130340f982a6f0bcc0b28b40 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ce4b1d786b674a8fa90b2b56dcb5cd1e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:36.980931 26599 tablet_service.cc:1467] Processing CreateTablet for tablet 536039b70452486684f7075cf45c1851 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ce4b1d786b674a8fa90b2b56dcb5cd1e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:36.981025 27403 tablet_bootstrap.cc:654] T 01b7ac587d134a61b333489d937c0a21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:36.981266 26600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4fa5ee9130340f982a6f0bcc0b28b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:36.984077 26599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 536039b70452486684f7075cf45c1851. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:36.994478 27403 tablet_bootstrap.cc:492] T 01b7ac587d134a61b333489d937c0a21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: No bootstrap required, opened a new log
I20240502 01:52:36.994679 27403 ts_tablet_manager.cc:1367] T 01b7ac587d134a61b333489d937c0a21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent bootstrapping tablet: real 0.016s	user 0.007s	sys 0.007s
I20240502 01:52:36.995528 27403 raft_consensus.cc:348] T 01b7ac587d134a61b333489d937c0a21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:36.995764 27403 raft_consensus.cc:374] T 01b7ac587d134a61b333489d937c0a21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:36.995851 27403 raft_consensus.cc:729] T 01b7ac587d134a61b333489d937c0a21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f215e8fdc8cb45f9a5eff7cb9e6ccfce, State: Initialized, Role: FOLLOWER
I20240502 01:52:36.996307 27405 tablet_bootstrap.cc:492] T d4fa5ee9130340f982a6f0bcc0b28b40 P 868e6e4358f4475bb45967b05eccb09f: Bootstrap starting.
I20240502 01:52:36.996217 27403 consensus_queue.cc:260] T 01b7ac587d134a61b333489d937c0a21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:36.997747 27403 ts_tablet_manager.cc:1398] T 01b7ac587d134a61b333489d937c0a21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.003s
I20240502 01:52:36.998127 27403 tablet_bootstrap.cc:492] T 536039b70452486684f7075cf45c1851 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap starting.
I20240502 01:52:36.998607 27405 tablet_bootstrap.cc:654] T d4fa5ee9130340f982a6f0bcc0b28b40 P 868e6e4358f4475bb45967b05eccb09f: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:37.001202 27403 tablet_bootstrap.cc:654] T 536039b70452486684f7075cf45c1851 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:37.007930 27405 tablet_bootstrap.cc:492] T d4fa5ee9130340f982a6f0bcc0b28b40 P 868e6e4358f4475bb45967b05eccb09f: No bootstrap required, opened a new log
I20240502 01:52:37.008150 27405 ts_tablet_manager.cc:1367] T d4fa5ee9130340f982a6f0bcc0b28b40 P 868e6e4358f4475bb45967b05eccb09f: Time spent bootstrapping tablet: real 0.012s	user 0.004s	sys 0.006s
I20240502 01:52:37.009097 27405 raft_consensus.cc:348] T d4fa5ee9130340f982a6f0bcc0b28b40 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } }
I20240502 01:52:37.009387 27405 raft_consensus.cc:374] T d4fa5ee9130340f982a6f0bcc0b28b40 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:37.009488 27405 raft_consensus.cc:729] T d4fa5ee9130340f982a6f0bcc0b28b40 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 868e6e4358f4475bb45967b05eccb09f, State: Initialized, Role: FOLLOWER
I20240502 01:52:37.009795 27405 consensus_queue.cc:260] T d4fa5ee9130340f982a6f0bcc0b28b40 P 868e6e4358f4475bb45967b05eccb09f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } }
I20240502 01:52:37.009938 27403 tablet_bootstrap.cc:492] T 536039b70452486684f7075cf45c1851 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: No bootstrap required, opened a new log
I20240502 01:52:37.010891 27403 ts_tablet_manager.cc:1367] T 536039b70452486684f7075cf45c1851 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent bootstrapping tablet: real 0.013s	user 0.001s	sys 0.009s
I20240502 01:52:37.011040 27405 ts_tablet_manager.cc:1398] T d4fa5ee9130340f982a6f0bcc0b28b40 P 868e6e4358f4475bb45967b05eccb09f: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 01:52:37.011363 27405 tablet_bootstrap.cc:492] T 536039b70452486684f7075cf45c1851 P 868e6e4358f4475bb45967b05eccb09f: Bootstrap starting.
I20240502 01:52:37.011948 27403 raft_consensus.cc:348] T 536039b70452486684f7075cf45c1851 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } }
I20240502 01:52:37.012288 27403 raft_consensus.cc:374] T 536039b70452486684f7075cf45c1851 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:37.012435 27403 raft_consensus.cc:729] T 536039b70452486684f7075cf45c1851 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f215e8fdc8cb45f9a5eff7cb9e6ccfce, State: Initialized, Role: FOLLOWER
I20240502 01:52:37.013402 27405 tablet_bootstrap.cc:654] T 536039b70452486684f7075cf45c1851 P 868e6e4358f4475bb45967b05eccb09f: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:37.013629 27403 consensus_queue.cc:260] T 536039b70452486684f7075cf45c1851 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } }
I20240502 01:52:37.014923 27403 ts_tablet_manager.cc:1398] T 536039b70452486684f7075cf45c1851 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 01:52:37.015465 27405 tablet_bootstrap.cc:492] T 536039b70452486684f7075cf45c1851 P 868e6e4358f4475bb45967b05eccb09f: No bootstrap required, opened a new log
I20240502 01:52:37.015648 27405 ts_tablet_manager.cc:1367] T 536039b70452486684f7075cf45c1851 P 868e6e4358f4475bb45967b05eccb09f: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240502 01:52:37.016520 27405 raft_consensus.cc:348] T 536039b70452486684f7075cf45c1851 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } }
I20240502 01:52:37.016767 27405 raft_consensus.cc:374] T 536039b70452486684f7075cf45c1851 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:37.016831 27405 raft_consensus.cc:729] T 536039b70452486684f7075cf45c1851 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 868e6e4358f4475bb45967b05eccb09f, State: Initialized, Role: FOLLOWER
I20240502 01:52:37.017113 27405 consensus_queue.cc:260] T 536039b70452486684f7075cf45c1851 P 868e6e4358f4475bb45967b05eccb09f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } }
I20240502 01:52:37.017797 27405 ts_tablet_manager.cc:1398] T 536039b70452486684f7075cf45c1851 P 868e6e4358f4475bb45967b05eccb09f: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 01:52:37.018146 27405 tablet_bootstrap.cc:492] T 01b7ac587d134a61b333489d937c0a21 P 868e6e4358f4475bb45967b05eccb09f: Bootstrap starting.
I20240502 01:52:37.019939 27405 tablet_bootstrap.cc:654] T 01b7ac587d134a61b333489d937c0a21 P 868e6e4358f4475bb45967b05eccb09f: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:37.023519 27408 tablet_bootstrap.cc:492] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Bootstrap starting.
I20240502 01:52:37.026239 27408 tablet_bootstrap.cc:654] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:37.027463 27402 tablet_bootstrap.cc:492] T 01b7ac587d134a61b333489d937c0a21 P d1e7d0eabafa46f382e19f596e2fc00d: Bootstrap starting.
I20240502 01:52:37.028334 27405 tablet_bootstrap.cc:492] T 01b7ac587d134a61b333489d937c0a21 P 868e6e4358f4475bb45967b05eccb09f: No bootstrap required, opened a new log
I20240502 01:52:37.028492 27405 ts_tablet_manager.cc:1367] T 01b7ac587d134a61b333489d937c0a21 P 868e6e4358f4475bb45967b05eccb09f: Time spent bootstrapping tablet: real 0.010s	user 0.004s	sys 0.005s
I20240502 01:52:37.028813 27408 tablet_bootstrap.cc:492] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: No bootstrap required, opened a new log
I20240502 01:52:37.028975 27408 ts_tablet_manager.cc:1367] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Time spent bootstrapping tablet: real 0.006s	user 0.003s	sys 0.000s
I20240502 01:52:37.029181 27405 raft_consensus.cc:348] T 01b7ac587d134a61b333489d937c0a21 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:37.029374 27405 raft_consensus.cc:374] T 01b7ac587d134a61b333489d937c0a21 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:37.029431 27402 tablet_bootstrap.cc:654] T 01b7ac587d134a61b333489d937c0a21 P d1e7d0eabafa46f382e19f596e2fc00d: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:37.029460 27405 raft_consensus.cc:729] T 01b7ac587d134a61b333489d937c0a21 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 868e6e4358f4475bb45967b05eccb09f, State: Initialized, Role: FOLLOWER
I20240502 01:52:37.029680 27405 consensus_queue.cc:260] T 01b7ac587d134a61b333489d937c0a21 P 868e6e4358f4475bb45967b05eccb09f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:37.029759 27408 raft_consensus.cc:348] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } }
I20240502 01:52:37.030179 27405 ts_tablet_manager.cc:1398] T 01b7ac587d134a61b333489d937c0a21 P 868e6e4358f4475bb45967b05eccb09f: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:52:37.029965 27408 raft_consensus.cc:374] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:37.030434 27408 raft_consensus.cc:729] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c70fbdc8a0b42ad8c09c6dbc842cc96, State: Initialized, Role: FOLLOWER
I20240502 01:52:37.030666 27408 consensus_queue.cc:260] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } }
I20240502 01:52:37.031244 27408 ts_tablet_manager.cc:1398] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:52:37.032366 27402 tablet_bootstrap.cc:492] T 01b7ac587d134a61b333489d937c0a21 P d1e7d0eabafa46f382e19f596e2fc00d: No bootstrap required, opened a new log
I20240502 01:52:37.032572 27402 ts_tablet_manager.cc:1367] T 01b7ac587d134a61b333489d937c0a21 P d1e7d0eabafa46f382e19f596e2fc00d: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240502 01:52:37.032936 27232 raft_consensus.cc:482] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:37.033078 27232 raft_consensus.cc:504] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } }
I20240502 01:52:37.033466 27402 raft_consensus.cc:348] T 01b7ac587d134a61b333489d937c0a21 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:37.034355 27402 raft_consensus.cc:374] T 01b7ac587d134a61b333489d937c0a21 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:37.034512 27232 leader_election.cc:288] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 868e6e4358f4475bb45967b05eccb09f (127.25.109.130:45571), d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132:37695)
I20240502 01:52:37.034520 27402 raft_consensus.cc:729] T 01b7ac587d134a61b333489d937c0a21 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e7d0eabafa46f382e19f596e2fc00d, State: Initialized, Role: FOLLOWER
I20240502 01:52:37.034793 26356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4fa5ee9130340f982a6f0bcc0b28b40" candidate_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "868e6e4358f4475bb45967b05eccb09f" is_pre_election: true
I20240502 01:52:37.034817 27402 consensus_queue.cc:260] T 01b7ac587d134a61b333489d937c0a21 P d1e7d0eabafa46f382e19f596e2fc00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:37.035053 26356 raft_consensus.cc:2454] T d4fa5ee9130340f982a6f0bcc0b28b40 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c70fbdc8a0b42ad8c09c6dbc842cc96 in term 0.
I20240502 01:52:37.035378 27402 ts_tablet_manager.cc:1398] T 01b7ac587d134a61b333489d937c0a21 P d1e7d0eabafa46f382e19f596e2fc00d: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.000s
I20240502 01:52:37.035478 26622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4fa5ee9130340f982a6f0bcc0b28b40" candidate_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" is_pre_election: true
I20240502 01:52:37.035688 27402 tablet_bootstrap.cc:492] T d4fa5ee9130340f982a6f0bcc0b28b40 P d1e7d0eabafa46f382e19f596e2fc00d: Bootstrap starting.
I20240502 01:52:37.036743 26423 leader_election.cc:302] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c70fbdc8a0b42ad8c09c6dbc842cc96, 868e6e4358f4475bb45967b05eccb09f; no voters: 
I20240502 01:52:37.037045 27232 raft_consensus.cc:2781] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:37.037194 27232 raft_consensus.cc:482] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:37.037295 27232 raft_consensus.cc:3037] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:37.037487 27402 tablet_bootstrap.cc:654] T d4fa5ee9130340f982a6f0bcc0b28b40 P d1e7d0eabafa46f382e19f596e2fc00d: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:37.039340 27402 tablet_bootstrap.cc:492] T d4fa5ee9130340f982a6f0bcc0b28b40 P d1e7d0eabafa46f382e19f596e2fc00d: No bootstrap required, opened a new log
I20240502 01:52:37.039459 27402 ts_tablet_manager.cc:1367] T d4fa5ee9130340f982a6f0bcc0b28b40 P d1e7d0eabafa46f382e19f596e2fc00d: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240502 01:52:37.039831 27232 raft_consensus.cc:504] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } }
W20240502 01:52:37.037205 26423 leader_election.cc:341] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132:37695): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:52:37.040212 27402 raft_consensus.cc:348] T d4fa5ee9130340f982a6f0bcc0b28b40 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } }
I20240502 01:52:37.040424 27402 raft_consensus.cc:374] T d4fa5ee9130340f982a6f0bcc0b28b40 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:37.040490 27232 leader_election.cc:288] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [CANDIDATE]: Term 1 election: Requested vote from peers 868e6e4358f4475bb45967b05eccb09f (127.25.109.130:45571), d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132:37695)
I20240502 01:52:37.040504 27402 raft_consensus.cc:729] T d4fa5ee9130340f982a6f0bcc0b28b40 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e7d0eabafa46f382e19f596e2fc00d, State: Initialized, Role: FOLLOWER
I20240502 01:52:37.040796 27402 consensus_queue.cc:260] T d4fa5ee9130340f982a6f0bcc0b28b40 P d1e7d0eabafa46f382e19f596e2fc00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } }
I20240502 01:52:37.041070 26622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4fa5ee9130340f982a6f0bcc0b28b40" candidate_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e7d0eabafa46f382e19f596e2fc00d"
I20240502 01:52:37.041086 26356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4fa5ee9130340f982a6f0bcc0b28b40" candidate_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "868e6e4358f4475bb45967b05eccb09f"
I20240502 01:52:37.041275 26622 raft_consensus.cc:3037] T d4fa5ee9130340f982a6f0bcc0b28b40 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:37.041309 26356 raft_consensus.cc:3037] T d4fa5ee9130340f982a6f0bcc0b28b40 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:37.041375 27402 ts_tablet_manager.cc:1398] T d4fa5ee9130340f982a6f0bcc0b28b40 P d1e7d0eabafa46f382e19f596e2fc00d: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:52:37.041639 27402 tablet_bootstrap.cc:492] T 536039b70452486684f7075cf45c1851 P d1e7d0eabafa46f382e19f596e2fc00d: Bootstrap starting.
I20240502 01:52:37.042883 27402 tablet_bootstrap.cc:654] T 536039b70452486684f7075cf45c1851 P d1e7d0eabafa46f382e19f596e2fc00d: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:37.044045 26356 raft_consensus.cc:2454] T d4fa5ee9130340f982a6f0bcc0b28b40 P 868e6e4358f4475bb45967b05eccb09f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c70fbdc8a0b42ad8c09c6dbc842cc96 in term 1.
I20240502 01:52:37.044065 26622 raft_consensus.cc:2454] T d4fa5ee9130340f982a6f0bcc0b28b40 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c70fbdc8a0b42ad8c09c6dbc842cc96 in term 1.
I20240502 01:52:37.044667 26423 leader_election.cc:302] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c70fbdc8a0b42ad8c09c6dbc842cc96, d1e7d0eabafa46f382e19f596e2fc00d; no voters: 
I20240502 01:52:37.044970 27232 raft_consensus.cc:2781] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:37.046305 27232 raft_consensus.cc:686] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 LEADER]: Becoming Leader. State: Replica: 6c70fbdc8a0b42ad8c09c6dbc842cc96, State: Running, Role: LEADER
I20240502 01:52:37.046689 27232 consensus_queue.cc:237] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } }
I20240502 01:52:37.046988 27402 tablet_bootstrap.cc:492] T 536039b70452486684f7075cf45c1851 P d1e7d0eabafa46f382e19f596e2fc00d: No bootstrap required, opened a new log
I20240502 01:52:37.047147 27402 ts_tablet_manager.cc:1367] T 536039b70452486684f7075cf45c1851 P d1e7d0eabafa46f382e19f596e2fc00d: Time spent bootstrapping tablet: real 0.006s	user 0.005s	sys 0.000s
I20240502 01:52:37.047937 27402 raft_consensus.cc:348] T 536039b70452486684f7075cf45c1851 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } }
I20240502 01:52:37.048969 27402 raft_consensus.cc:374] T 536039b70452486684f7075cf45c1851 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:37.049062 27402 raft_consensus.cc:729] T 536039b70452486684f7075cf45c1851 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e7d0eabafa46f382e19f596e2fc00d, State: Initialized, Role: FOLLOWER
I20240502 01:52:37.049268 27402 consensus_queue.cc:260] T 536039b70452486684f7075cf45c1851 P d1e7d0eabafa46f382e19f596e2fc00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } }
I20240502 01:52:37.049881 27402 ts_tablet_manager.cc:1398] T 536039b70452486684f7075cf45c1851 P d1e7d0eabafa46f382e19f596e2fc00d: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.000s
I20240502 01:52:37.049886 26075 catalog_manager.cc:5484] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131). New cstate: current_term: 1 leader_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:37.133510 27233 raft_consensus.cc:482] T 01b7ac587d134a61b333489d937c0a21 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:37.133742 27233 raft_consensus.cc:504] T 01b7ac587d134a61b333489d937c0a21 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:37.134564 27233 leader_election.cc:288] T 01b7ac587d134a61b333489d937c0a21 P d1e7d0eabafa46f382e19f596e2fc00d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541), 868e6e4358f4475bb45967b05eccb09f (127.25.109.130:45571)
I20240502 01:52:37.135098 26223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b7ac587d134a61b333489d937c0a21" candidate_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" is_pre_election: true
I20240502 01:52:37.135169 26356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b7ac587d134a61b333489d937c0a21" candidate_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "868e6e4358f4475bb45967b05eccb09f" is_pre_election: true
I20240502 01:52:37.135389 26223 raft_consensus.cc:2454] T 01b7ac587d134a61b333489d937c0a21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e7d0eabafa46f382e19f596e2fc00d in term 0.
I20240502 01:52:37.135403 26356 raft_consensus.cc:2454] T 01b7ac587d134a61b333489d937c0a21 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e7d0eabafa46f382e19f596e2fc00d in term 0.
I20240502 01:52:37.135989 26556 leader_election.cc:302] T 01b7ac587d134a61b333489d937c0a21 P d1e7d0eabafa46f382e19f596e2fc00d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 868e6e4358f4475bb45967b05eccb09f, d1e7d0eabafa46f382e19f596e2fc00d, f215e8fdc8cb45f9a5eff7cb9e6ccfce; no voters: 
I20240502 01:52:37.136312 27233 raft_consensus.cc:2781] T 01b7ac587d134a61b333489d937c0a21 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:37.136431 27233 raft_consensus.cc:482] T 01b7ac587d134a61b333489d937c0a21 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:37.136523 27233 raft_consensus.cc:3037] T 01b7ac587d134a61b333489d937c0a21 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:37.138485 27233 raft_consensus.cc:504] T 01b7ac587d134a61b333489d937c0a21 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:37.139106 27233 leader_election.cc:288] T 01b7ac587d134a61b333489d937c0a21 P d1e7d0eabafa46f382e19f596e2fc00d [CANDIDATE]: Term 1 election: Requested vote from peers f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541), 868e6e4358f4475bb45967b05eccb09f (127.25.109.130:45571)
I20240502 01:52:37.139618 26223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b7ac587d134a61b333489d937c0a21" candidate_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce"
I20240502 01:52:37.139755 26356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b7ac587d134a61b333489d937c0a21" candidate_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "868e6e4358f4475bb45967b05eccb09f"
I20240502 01:52:37.139842 26223 raft_consensus.cc:3037] T 01b7ac587d134a61b333489d937c0a21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:37.139995 26356 raft_consensus.cc:3037] T 01b7ac587d134a61b333489d937c0a21 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:37.141783 26223 raft_consensus.cc:2454] T 01b7ac587d134a61b333489d937c0a21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e7d0eabafa46f382e19f596e2fc00d in term 1.
I20240502 01:52:37.141878 26356 raft_consensus.cc:2454] T 01b7ac587d134a61b333489d937c0a21 P 868e6e4358f4475bb45967b05eccb09f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e7d0eabafa46f382e19f596e2fc00d in term 1.
I20240502 01:52:37.142251 26556 leader_election.cc:302] T 01b7ac587d134a61b333489d937c0a21 P d1e7d0eabafa46f382e19f596e2fc00d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e7d0eabafa46f382e19f596e2fc00d, f215e8fdc8cb45f9a5eff7cb9e6ccfce; no voters: 
I20240502 01:52:37.142562 27233 raft_consensus.cc:2781] T 01b7ac587d134a61b333489d937c0a21 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:37.142980 27233 raft_consensus.cc:686] T 01b7ac587d134a61b333489d937c0a21 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 LEADER]: Becoming Leader. State: Replica: d1e7d0eabafa46f382e19f596e2fc00d, State: Running, Role: LEADER
I20240502 01:52:37.143249 27233 consensus_queue.cc:237] T 01b7ac587d134a61b333489d937c0a21 P d1e7d0eabafa46f382e19f596e2fc00d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:37.146162 26077 catalog_manager.cc:5484] T 01b7ac587d134a61b333489d937c0a21 P d1e7d0eabafa46f382e19f596e2fc00d reported cstate change: term changed from 0 to 1, leader changed from <none> to d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132). New cstate: current_term: 1 leader_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:37.347106 27233 raft_consensus.cc:482] T 536039b70452486684f7075cf45c1851 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:37.347318 27233 raft_consensus.cc:504] T 536039b70452486684f7075cf45c1851 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } }
I20240502 01:52:37.348068 27233 leader_election.cc:288] T 536039b70452486684f7075cf45c1851 P d1e7d0eabafa46f382e19f596e2fc00d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 868e6e4358f4475bb45967b05eccb09f (127.25.109.130:45571), f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541)
I20240502 01:52:37.348615 26356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "536039b70452486684f7075cf45c1851" candidate_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "868e6e4358f4475bb45967b05eccb09f" is_pre_election: true
I20240502 01:52:37.348650 26223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "536039b70452486684f7075cf45c1851" candidate_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" is_pre_election: true
I20240502 01:52:37.348855 26223 raft_consensus.cc:2454] T 536039b70452486684f7075cf45c1851 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e7d0eabafa46f382e19f596e2fc00d in term 0.
I20240502 01:52:37.348861 26356 raft_consensus.cc:2454] T 536039b70452486684f7075cf45c1851 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e7d0eabafa46f382e19f596e2fc00d in term 0.
I20240502 01:52:37.349313 26555 leader_election.cc:302] T 536039b70452486684f7075cf45c1851 P d1e7d0eabafa46f382e19f596e2fc00d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 868e6e4358f4475bb45967b05eccb09f, d1e7d0eabafa46f382e19f596e2fc00d, f215e8fdc8cb45f9a5eff7cb9e6ccfce; no voters: 
I20240502 01:52:37.349682 27233 raft_consensus.cc:2781] T 536039b70452486684f7075cf45c1851 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:37.349794 27233 raft_consensus.cc:482] T 536039b70452486684f7075cf45c1851 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:37.349900 27233 raft_consensus.cc:3037] T 536039b70452486684f7075cf45c1851 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:37.353152 27233 raft_consensus.cc:504] T 536039b70452486684f7075cf45c1851 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } }
I20240502 01:52:37.353863 27233 leader_election.cc:288] T 536039b70452486684f7075cf45c1851 P d1e7d0eabafa46f382e19f596e2fc00d [CANDIDATE]: Term 1 election: Requested vote from peers 868e6e4358f4475bb45967b05eccb09f (127.25.109.130:45571), f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541)
I20240502 01:52:37.354427 26356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "536039b70452486684f7075cf45c1851" candidate_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "868e6e4358f4475bb45967b05eccb09f"
I20240502 01:52:37.354494 26223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "536039b70452486684f7075cf45c1851" candidate_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce"
I20240502 01:52:37.354666 26356 raft_consensus.cc:3037] T 536039b70452486684f7075cf45c1851 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:37.354712 26223 raft_consensus.cc:3037] T 536039b70452486684f7075cf45c1851 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:37.357085 26356 raft_consensus.cc:2454] T 536039b70452486684f7075cf45c1851 P 868e6e4358f4475bb45967b05eccb09f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e7d0eabafa46f382e19f596e2fc00d in term 1.
I20240502 01:52:37.357594 26555 leader_election.cc:302] T 536039b70452486684f7075cf45c1851 P d1e7d0eabafa46f382e19f596e2fc00d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 868e6e4358f4475bb45967b05eccb09f, d1e7d0eabafa46f382e19f596e2fc00d; no voters: 
I20240502 01:52:37.357733 26223 raft_consensus.cc:2454] T 536039b70452486684f7075cf45c1851 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e7d0eabafa46f382e19f596e2fc00d in term 1.
I20240502 01:52:37.357913 27233 raft_consensus.cc:2781] T 536039b70452486684f7075cf45c1851 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:37.358116 27233 raft_consensus.cc:686] T 536039b70452486684f7075cf45c1851 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 LEADER]: Becoming Leader. State: Replica: d1e7d0eabafa46f382e19f596e2fc00d, State: Running, Role: LEADER
I20240502 01:52:37.358433 27233 consensus_queue.cc:237] T 536039b70452486684f7075cf45c1851 P d1e7d0eabafa46f382e19f596e2fc00d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } }
I20240502 01:52:37.361503 26077 catalog_manager.cc:5484] T 536039b70452486684f7075cf45c1851 P d1e7d0eabafa46f382e19f596e2fc00d reported cstate change: term changed from 0 to 1, leader changed from <none> to d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132). New cstate: current_term: 1 leader_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:37.439986 27232 consensus_queue.cc:1035] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:37.444540 27232 consensus_queue.cc:1035] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:37.681798 27233 consensus_queue.cc:1035] T 01b7ac587d134a61b333489d937c0a21 P d1e7d0eabafa46f382e19f596e2fc00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:37.688669 27417 consensus_queue.cc:1035] T 01b7ac587d134a61b333489d937c0a21 P d1e7d0eabafa46f382e19f596e2fc00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:52:37.798352 27425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:37.798686 27425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:37.806319 27425 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:52:37.838698 27233 consensus_queue.cc:1035] T 536039b70452486684f7075cf45c1851 P d1e7d0eabafa46f382e19f596e2fc00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:37.843072 27440 consensus_queue.cc:1035] T 536039b70452486684f7075cf45c1851 P d1e7d0eabafa46f382e19f596e2fc00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'

INSERT report
    rows total: 2000
    time total: 224.953 ms
  time per row: 0.112477 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      2        596 libcrypto.so.1.1
-----------------------------------------------------

I20240502 01:52:39.053395 26077 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57636:
name: "rebalance_test_table_3"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:52:39.063500 26467 tablet_service.cc:1467] Processing CreateTablet for tablet fbd456fceb0f4c6db45662eb619cfd21 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3dba071b1adb47bebabd07a94b201ce1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:39.064019 26467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbd456fceb0f4c6db45662eb619cfd21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:39.065199 26336 tablet_service.cc:1467] Processing CreateTablet for tablet fbd456fceb0f4c6db45662eb619cfd21 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3dba071b1adb47bebabd07a94b201ce1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:39.065326 26599 tablet_service.cc:1467] Processing CreateTablet for tablet 38dbb02fca694854ab66393a87c84908 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3dba071b1adb47bebabd07a94b201ce1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:39.065325 26601 tablet_service.cc:1467] Processing CreateTablet for tablet ebe8589feed3400aa6bca73df420778d (DEFAULT_TABLE table=rebalance_test_table_3 [id=3dba071b1adb47bebabd07a94b201ce1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:39.065670 26336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbd456fceb0f4c6db45662eb619cfd21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:39.065852 26599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38dbb02fca694854ab66393a87c84908. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:39.066076 26601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebe8589feed3400aa6bca73df420778d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:39.069659 26468 tablet_service.cc:1467] Processing CreateTablet for tablet ebe8589feed3400aa6bca73df420778d (DEFAULT_TABLE table=rebalance_test_table_3 [id=3dba071b1adb47bebabd07a94b201ce1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:39.069676 26335 tablet_service.cc:1467] Processing CreateTablet for tablet 38dbb02fca694854ab66393a87c84908 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3dba071b1adb47bebabd07a94b201ce1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:39.070132 26335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38dbb02fca694854ab66393a87c84908. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:39.070132 26468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebe8589feed3400aa6bca73df420778d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:39.076462 26469 tablet_service.cc:1467] Processing CreateTablet for tablet 38dbb02fca694854ab66393a87c84908 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3dba071b1adb47bebabd07a94b201ce1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:39.077032 26469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38dbb02fca694854ab66393a87c84908. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:39.080302 26204 tablet_service.cc:1467] Processing CreateTablet for tablet ebe8589feed3400aa6bca73df420778d (DEFAULT_TABLE table=rebalance_test_table_3 [id=3dba071b1adb47bebabd07a94b201ce1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:39.080843 26204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebe8589feed3400aa6bca73df420778d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:39.081202 26202 tablet_service.cc:1467] Processing CreateTablet for tablet fbd456fceb0f4c6db45662eb619cfd21 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3dba071b1adb47bebabd07a94b201ce1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:39.082137 27489 tablet_bootstrap.cc:492] T ebe8589feed3400aa6bca73df420778d P d1e7d0eabafa46f382e19f596e2fc00d: Bootstrap starting.
I20240502 01:52:39.083531 27489 tablet_bootstrap.cc:654] T ebe8589feed3400aa6bca73df420778d P d1e7d0eabafa46f382e19f596e2fc00d: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:39.087478 26202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbd456fceb0f4c6db45662eb619cfd21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:39.087769 27491 tablet_bootstrap.cc:492] T ebe8589feed3400aa6bca73df420778d P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap starting.
I20240502 01:52:39.088315 27489 tablet_bootstrap.cc:492] T ebe8589feed3400aa6bca73df420778d P d1e7d0eabafa46f382e19f596e2fc00d: No bootstrap required, opened a new log
I20240502 01:52:39.088488 27489 ts_tablet_manager.cc:1367] T ebe8589feed3400aa6bca73df420778d P d1e7d0eabafa46f382e19f596e2fc00d: Time spent bootstrapping tablet: real 0.006s	user 0.003s	sys 0.000s
I20240502 01:52:39.089964 27491 tablet_bootstrap.cc:654] T ebe8589feed3400aa6bca73df420778d P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:39.092396 27489 raft_consensus.cc:348] T ebe8589feed3400aa6bca73df420778d P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } }
I20240502 01:52:39.093139 27489 raft_consensus.cc:374] T ebe8589feed3400aa6bca73df420778d P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:39.093299 27489 raft_consensus.cc:729] T ebe8589feed3400aa6bca73df420778d P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e7d0eabafa46f382e19f596e2fc00d, State: Initialized, Role: FOLLOWER
I20240502 01:52:39.093657 27489 consensus_queue.cc:260] T ebe8589feed3400aa6bca73df420778d P d1e7d0eabafa46f382e19f596e2fc00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } }
I20240502 01:52:39.094403 27489 ts_tablet_manager.cc:1398] T ebe8589feed3400aa6bca73df420778d P d1e7d0eabafa46f382e19f596e2fc00d: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.000s
I20240502 01:52:39.094789 27489 tablet_bootstrap.cc:492] T 38dbb02fca694854ab66393a87c84908 P d1e7d0eabafa46f382e19f596e2fc00d: Bootstrap starting.
I20240502 01:52:39.096834 27489 tablet_bootstrap.cc:654] T 38dbb02fca694854ab66393a87c84908 P d1e7d0eabafa46f382e19f596e2fc00d: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:39.104097 27494 tablet_bootstrap.cc:492] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f: Bootstrap starting.
I20240502 01:52:39.105729 27488 tablet_bootstrap.cc:492] T fbd456fceb0f4c6db45662eb619cfd21 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Bootstrap starting.
I20240502 01:52:39.105973 27494 tablet_bootstrap.cc:654] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:39.107081 27489 tablet_bootstrap.cc:492] T 38dbb02fca694854ab66393a87c84908 P d1e7d0eabafa46f382e19f596e2fc00d: No bootstrap required, opened a new log
I20240502 01:52:39.107431 27489 ts_tablet_manager.cc:1367] T 38dbb02fca694854ab66393a87c84908 P d1e7d0eabafa46f382e19f596e2fc00d: Time spent bootstrapping tablet: real 0.013s	user 0.011s	sys 0.000s
I20240502 01:52:39.107482 27491 tablet_bootstrap.cc:492] T ebe8589feed3400aa6bca73df420778d P f215e8fdc8cb45f9a5eff7cb9e6ccfce: No bootstrap required, opened a new log
I20240502 01:52:39.107694 27491 ts_tablet_manager.cc:1367] T ebe8589feed3400aa6bca73df420778d P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent bootstrapping tablet: real 0.020s	user 0.006s	sys 0.000s
I20240502 01:52:39.108404 27489 raft_consensus.cc:348] T 38dbb02fca694854ab66393a87c84908 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:52:39.108366 27488 tablet_bootstrap.cc:654] T fbd456fceb0f4c6db45662eb619cfd21 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:39.108754 27489 raft_consensus.cc:374] T 38dbb02fca694854ab66393a87c84908 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:39.108901 27489 raft_consensus.cc:729] T 38dbb02fca694854ab66393a87c84908 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e7d0eabafa46f382e19f596e2fc00d, State: Initialized, Role: FOLLOWER
I20240502 01:52:39.109256 27489 consensus_queue.cc:260] T 38dbb02fca694854ab66393a87c84908 P d1e7d0eabafa46f382e19f596e2fc00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:52:39.110468 27491 raft_consensus.cc:348] T ebe8589feed3400aa6bca73df420778d P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } }
I20240502 01:52:39.110622 27489 ts_tablet_manager.cc:1398] T 38dbb02fca694854ab66393a87c84908 P d1e7d0eabafa46f382e19f596e2fc00d: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:52:39.110774 27491 raft_consensus.cc:374] T ebe8589feed3400aa6bca73df420778d P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:39.110932 27491 raft_consensus.cc:729] T ebe8589feed3400aa6bca73df420778d P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f215e8fdc8cb45f9a5eff7cb9e6ccfce, State: Initialized, Role: FOLLOWER
I20240502 01:52:39.111220 27491 consensus_queue.cc:260] T ebe8589feed3400aa6bca73df420778d P f215e8fdc8cb45f9a5eff7cb9e6ccfce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } }
I20240502 01:52:39.111912 27491 ts_tablet_manager.cc:1398] T ebe8589feed3400aa6bca73df420778d P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.002s
I20240502 01:52:39.113479 27488 tablet_bootstrap.cc:492] T fbd456fceb0f4c6db45662eb619cfd21 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: No bootstrap required, opened a new log
I20240502 01:52:39.113714 27488 ts_tablet_manager.cc:1367] T fbd456fceb0f4c6db45662eb619cfd21 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Time spent bootstrapping tablet: real 0.008s	user 0.003s	sys 0.004s
I20240502 01:52:39.114053 27491 tablet_bootstrap.cc:492] T fbd456fceb0f4c6db45662eb619cfd21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap starting.
I20240502 01:52:39.114145 27494 tablet_bootstrap.cc:492] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f: No bootstrap required, opened a new log
I20240502 01:52:39.114439 27494 ts_tablet_manager.cc:1367] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f: Time spent bootstrapping tablet: real 0.010s	user 0.005s	sys 0.000s
I20240502 01:52:39.114838 27488 raft_consensus.cc:348] T fbd456fceb0f4c6db45662eb619cfd21 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:39.115147 27488 raft_consensus.cc:374] T fbd456fceb0f4c6db45662eb619cfd21 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:39.115300 27488 raft_consensus.cc:729] T fbd456fceb0f4c6db45662eb619cfd21 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c70fbdc8a0b42ad8c09c6dbc842cc96, State: Initialized, Role: FOLLOWER
I20240502 01:52:39.115643 27488 consensus_queue.cc:260] T fbd456fceb0f4c6db45662eb619cfd21 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:39.115983 27494 raft_consensus.cc:348] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:52:39.116232 27491 tablet_bootstrap.cc:654] T fbd456fceb0f4c6db45662eb619cfd21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:39.116257 27494 raft_consensus.cc:374] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:39.116466 27494 raft_consensus.cc:729] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 868e6e4358f4475bb45967b05eccb09f, State: Initialized, Role: FOLLOWER
I20240502 01:52:39.116801 27494 consensus_queue.cc:260] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:52:39.117450 27494 ts_tablet_manager.cc:1398] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:52:39.118608 27494 tablet_bootstrap.cc:492] T fbd456fceb0f4c6db45662eb619cfd21 P 868e6e4358f4475bb45967b05eccb09f: Bootstrap starting.
I20240502 01:52:39.118681 27488 ts_tablet_manager.cc:1398] T fbd456fceb0f4c6db45662eb619cfd21 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.002s
I20240502 01:52:39.120363 27494 tablet_bootstrap.cc:654] T fbd456fceb0f4c6db45662eb619cfd21 P 868e6e4358f4475bb45967b05eccb09f: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:39.120883 27488 tablet_bootstrap.cc:492] T ebe8589feed3400aa6bca73df420778d P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Bootstrap starting.
I20240502 01:52:39.123759 27488 tablet_bootstrap.cc:654] T ebe8589feed3400aa6bca73df420778d P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:39.124305 27491 tablet_bootstrap.cc:492] T fbd456fceb0f4c6db45662eb619cfd21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: No bootstrap required, opened a new log
I20240502 01:52:39.124483 27491 ts_tablet_manager.cc:1367] T fbd456fceb0f4c6db45662eb619cfd21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent bootstrapping tablet: real 0.010s	user 0.003s	sys 0.006s
I20240502 01:52:39.125139 27491 raft_consensus.cc:348] T fbd456fceb0f4c6db45662eb619cfd21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:39.125345 27491 raft_consensus.cc:374] T fbd456fceb0f4c6db45662eb619cfd21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:39.125419 27491 raft_consensus.cc:729] T fbd456fceb0f4c6db45662eb619cfd21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f215e8fdc8cb45f9a5eff7cb9e6ccfce, State: Initialized, Role: FOLLOWER
I20240502 01:52:39.125437 27488 tablet_bootstrap.cc:492] T ebe8589feed3400aa6bca73df420778d P 6c70fbdc8a0b42ad8c09c6dbc842cc96: No bootstrap required, opened a new log
I20240502 01:52:39.125597 27488 ts_tablet_manager.cc:1367] T ebe8589feed3400aa6bca73df420778d P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Time spent bootstrapping tablet: real 0.005s	user 0.002s	sys 0.002s
I20240502 01:52:39.125681 27491 consensus_queue.cc:260] T fbd456fceb0f4c6db45662eb619cfd21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:39.126199 27491 ts_tablet_manager.cc:1398] T fbd456fceb0f4c6db45662eb619cfd21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:52:39.126303 27488 raft_consensus.cc:348] T ebe8589feed3400aa6bca73df420778d P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } }
I20240502 01:52:39.126533 27488 raft_consensus.cc:374] T ebe8589feed3400aa6bca73df420778d P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:39.126616 27488 raft_consensus.cc:729] T ebe8589feed3400aa6bca73df420778d P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c70fbdc8a0b42ad8c09c6dbc842cc96, State: Initialized, Role: FOLLOWER
I20240502 01:52:39.126868 27488 consensus_queue.cc:260] T ebe8589feed3400aa6bca73df420778d P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } }
I20240502 01:52:39.127094 27494 tablet_bootstrap.cc:492] T fbd456fceb0f4c6db45662eb619cfd21 P 868e6e4358f4475bb45967b05eccb09f: No bootstrap required, opened a new log
I20240502 01:52:39.127270 27494 ts_tablet_manager.cc:1367] T fbd456fceb0f4c6db45662eb619cfd21 P 868e6e4358f4475bb45967b05eccb09f: Time spent bootstrapping tablet: real 0.009s	user 0.005s	sys 0.000s
I20240502 01:52:39.127941 27494 raft_consensus.cc:348] T fbd456fceb0f4c6db45662eb619cfd21 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:39.128149 27494 raft_consensus.cc:374] T fbd456fceb0f4c6db45662eb619cfd21 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:39.128175 27488 ts_tablet_manager.cc:1398] T ebe8589feed3400aa6bca73df420778d P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:52:39.128227 27494 raft_consensus.cc:729] T fbd456fceb0f4c6db45662eb619cfd21 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 868e6e4358f4475bb45967b05eccb09f, State: Initialized, Role: FOLLOWER
I20240502 01:52:39.128511 27488 tablet_bootstrap.cc:492] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Bootstrap starting.
I20240502 01:52:39.128461 27494 consensus_queue.cc:260] T fbd456fceb0f4c6db45662eb619cfd21 P 868e6e4358f4475bb45967b05eccb09f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:39.128968 27494 ts_tablet_manager.cc:1398] T fbd456fceb0f4c6db45662eb619cfd21 P 868e6e4358f4475bb45967b05eccb09f: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:52:39.130112 27488 tablet_bootstrap.cc:654] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:39.133008 27488 tablet_bootstrap.cc:492] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: No bootstrap required, opened a new log
I20240502 01:52:39.133176 27488 ts_tablet_manager.cc:1367] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Time spent bootstrapping tablet: real 0.005s	user 0.002s	sys 0.002s
I20240502 01:52:39.133965 27488 raft_consensus.cc:348] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:52:39.134166 27488 raft_consensus.cc:374] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:39.134259 27488 raft_consensus.cc:729] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c70fbdc8a0b42ad8c09c6dbc842cc96, State: Initialized, Role: FOLLOWER
I20240502 01:52:39.134466 27488 consensus_queue.cc:260] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:52:39.134915 27488 ts_tablet_manager.cc:1398] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:52:39.142764 27475 raft_consensus.cc:482] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:39.142933 27475 raft_consensus.cc:504] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:52:39.143575 27475 leader_election.cc:288] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 868e6e4358f4475bb45967b05eccb09f (127.25.109.130:45571), d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132:37695)
I20240502 01:52:39.144150 26357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbb02fca694854ab66393a87c84908" candidate_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "868e6e4358f4475bb45967b05eccb09f" is_pre_election: true
I20240502 01:52:39.144213 26622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbb02fca694854ab66393a87c84908" candidate_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" is_pre_election: true
I20240502 01:52:39.144393 26357 raft_consensus.cc:2454] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c70fbdc8a0b42ad8c09c6dbc842cc96 in term 0.
I20240502 01:52:39.144481 26622 raft_consensus.cc:2454] T 38dbb02fca694854ab66393a87c84908 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c70fbdc8a0b42ad8c09c6dbc842cc96 in term 0.
I20240502 01:52:39.144825 26423 leader_election.cc:302] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c70fbdc8a0b42ad8c09c6dbc842cc96, 868e6e4358f4475bb45967b05eccb09f; no voters: 
I20240502 01:52:39.145100 27475 raft_consensus.cc:2781] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:39.145212 27475 raft_consensus.cc:482] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:39.145289 27475 raft_consensus.cc:3037] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:39.205901 27475 raft_consensus.cc:504] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:52:39.207177 27475 leader_election.cc:288] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [CANDIDATE]: Term 1 election: Requested vote from peers 868e6e4358f4475bb45967b05eccb09f (127.25.109.130:45571), d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132:37695)
I20240502 01:52:39.207332 26357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbb02fca694854ab66393a87c84908" candidate_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "868e6e4358f4475bb45967b05eccb09f"
I20240502 01:52:39.207612 26357 raft_consensus.cc:3037] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:39.208199 26622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbb02fca694854ab66393a87c84908" candidate_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e7d0eabafa46f382e19f596e2fc00d"
I20240502 01:52:39.208469 26622 raft_consensus.cc:3037] T 38dbb02fca694854ab66393a87c84908 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:39.259243 27388 raft_consensus.cc:482] T ebe8589feed3400aa6bca73df420778d P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:39.259449 27388 raft_consensus.cc:504] T ebe8589feed3400aa6bca73df420778d P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } }
I20240502 01:52:39.260174 27388 leader_election.cc:288] T ebe8589feed3400aa6bca73df420778d P f215e8fdc8cb45f9a5eff7cb9e6ccfce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132:37695), 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131:38693)
I20240502 01:52:39.260885 26488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebe8589feed3400aa6bca73df420778d" candidate_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" is_pre_election: true
I20240502 01:52:39.260890 26621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebe8589feed3400aa6bca73df420778d" candidate_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" is_pre_election: true
I20240502 01:52:39.261161 26488 raft_consensus.cc:2454] T ebe8589feed3400aa6bca73df420778d P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f215e8fdc8cb45f9a5eff7cb9e6ccfce in term 0.
I20240502 01:52:39.261161 26621 raft_consensus.cc:2454] T ebe8589feed3400aa6bca73df420778d P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f215e8fdc8cb45f9a5eff7cb9e6ccfce in term 0.
I20240502 01:52:39.261641 26157 leader_election.cc:302] T ebe8589feed3400aa6bca73df420778d P f215e8fdc8cb45f9a5eff7cb9e6ccfce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e7d0eabafa46f382e19f596e2fc00d, f215e8fdc8cb45f9a5eff7cb9e6ccfce; no voters: 
I20240502 01:52:39.261960 27388 raft_consensus.cc:2781] T ebe8589feed3400aa6bca73df420778d P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:39.262089 27388 raft_consensus.cc:482] T ebe8589feed3400aa6bca73df420778d P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:39.262183 27388 raft_consensus.cc:3037] T ebe8589feed3400aa6bca73df420778d P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:39.270834 27503 raft_consensus.cc:482] T fbd456fceb0f4c6db45662eb619cfd21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:39.271028 27503 raft_consensus.cc:504] T fbd456fceb0f4c6db45662eb619cfd21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:39.271951 27503 leader_election.cc:288] T fbd456fceb0f4c6db45662eb619cfd21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131:38693), 868e6e4358f4475bb45967b05eccb09f (127.25.109.130:45571)
I20240502 01:52:39.272665 26488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbd456fceb0f4c6db45662eb619cfd21" candidate_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" is_pre_election: true
I20240502 01:52:39.272962 26488 raft_consensus.cc:2454] T fbd456fceb0f4c6db45662eb619cfd21 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f215e8fdc8cb45f9a5eff7cb9e6ccfce in term 0.
I20240502 01:52:39.273413 26158 leader_election.cc:302] T fbd456fceb0f4c6db45662eb619cfd21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c70fbdc8a0b42ad8c09c6dbc842cc96, f215e8fdc8cb45f9a5eff7cb9e6ccfce; no voters: 
I20240502 01:52:39.273725 27503 raft_consensus.cc:2781] T fbd456fceb0f4c6db45662eb619cfd21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:39.273965 27503 raft_consensus.cc:482] T fbd456fceb0f4c6db45662eb619cfd21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:39.274171 27503 raft_consensus.cc:3037] T fbd456fceb0f4c6db45662eb619cfd21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:39.278798 26356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbd456fceb0f4c6db45662eb619cfd21" candidate_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "868e6e4358f4475bb45967b05eccb09f" is_pre_election: true
I20240502 01:52:39.279057 26356 raft_consensus.cc:2454] T fbd456fceb0f4c6db45662eb619cfd21 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f215e8fdc8cb45f9a5eff7cb9e6ccfce in term 0.
I20240502 01:52:39.306816 26357 raft_consensus.cc:2454] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c70fbdc8a0b42ad8c09c6dbc842cc96 in term 1.
I20240502 01:52:39.306816 26622 raft_consensus.cc:2454] T 38dbb02fca694854ab66393a87c84908 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c70fbdc8a0b42ad8c09c6dbc842cc96 in term 1.
I20240502 01:52:39.306885 27388 raft_consensus.cc:504] T ebe8589feed3400aa6bca73df420778d P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } }
I20240502 01:52:39.307631 26423 leader_election.cc:302] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c70fbdc8a0b42ad8c09c6dbc842cc96, d1e7d0eabafa46f382e19f596e2fc00d; no voters: 
I20240502 01:52:39.307796 27388 leader_election.cc:288] T ebe8589feed3400aa6bca73df420778d P f215e8fdc8cb45f9a5eff7cb9e6ccfce [CANDIDATE]: Term 1 election: Requested vote from peers d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132:37695), 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131:38693)
I20240502 01:52:39.308332 27475 raft_consensus.cc:2781] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:39.308495 27475 raft_consensus.cc:686] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 LEADER]: Becoming Leader. State: Replica: 6c70fbdc8a0b42ad8c09c6dbc842cc96, State: Running, Role: LEADER
I20240502 01:52:39.308674 26488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebe8589feed3400aa6bca73df420778d" candidate_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96"
I20240502 01:52:39.308892 26488 raft_consensus.cc:3037] T ebe8589feed3400aa6bca73df420778d P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:39.308815 27475 consensus_queue.cc:237] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:52:39.311013 26622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebe8589feed3400aa6bca73df420778d" candidate_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e7d0eabafa46f382e19f596e2fc00d"
I20240502 01:52:39.310997 27503 raft_consensus.cc:504] T fbd456fceb0f4c6db45662eb619cfd21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:39.311249 26622 raft_consensus.cc:3037] T ebe8589feed3400aa6bca73df420778d P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:39.311934 27503 leader_election.cc:288] T fbd456fceb0f4c6db45662eb619cfd21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [CANDIDATE]: Term 1 election: Requested vote from peers 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131:38693), 868e6e4358f4475bb45967b05eccb09f (127.25.109.130:45571)
I20240502 01:52:39.312507 26489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbd456fceb0f4c6db45662eb619cfd21" candidate_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96"
I20240502 01:52:39.312727 26489 raft_consensus.cc:3037] T fbd456fceb0f4c6db45662eb619cfd21 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:39.313472 26357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbd456fceb0f4c6db45662eb619cfd21" candidate_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "868e6e4358f4475bb45967b05eccb09f"
I20240502 01:52:39.313787 26357 raft_consensus.cc:3037] T fbd456fceb0f4c6db45662eb619cfd21 P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:39.314306 26077 catalog_manager.cc:5484] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131). New cstate: current_term: 1 leader_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:39.409103 26488 raft_consensus.cc:2454] T ebe8589feed3400aa6bca73df420778d P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f215e8fdc8cb45f9a5eff7cb9e6ccfce in term 1.
I20240502 01:52:39.409103 26622 raft_consensus.cc:2454] T ebe8589feed3400aa6bca73df420778d P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f215e8fdc8cb45f9a5eff7cb9e6ccfce in term 1.
I20240502 01:52:39.409111 26357 raft_consensus.cc:2454] T fbd456fceb0f4c6db45662eb619cfd21 P 868e6e4358f4475bb45967b05eccb09f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f215e8fdc8cb45f9a5eff7cb9e6ccfce in term 1.
I20240502 01:52:39.409739 26157 leader_election.cc:302] T fbd456fceb0f4c6db45662eb619cfd21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 868e6e4358f4475bb45967b05eccb09f, f215e8fdc8cb45f9a5eff7cb9e6ccfce; no voters: 
I20240502 01:52:39.410040 26158 leader_election.cc:302] T ebe8589feed3400aa6bca73df420778d P f215e8fdc8cb45f9a5eff7cb9e6ccfce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c70fbdc8a0b42ad8c09c6dbc842cc96, f215e8fdc8cb45f9a5eff7cb9e6ccfce; no voters: 
I20240502 01:52:39.410404 27388 raft_consensus.cc:2781] T fbd456fceb0f4c6db45662eb619cfd21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:39.410724 27503 raft_consensus.cc:2781] T ebe8589feed3400aa6bca73df420778d P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:39.412235 26489 raft_consensus.cc:2454] T fbd456fceb0f4c6db45662eb619cfd21 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f215e8fdc8cb45f9a5eff7cb9e6ccfce in term 1.
I20240502 01:52:39.413319 27503 raft_consensus.cc:686] T ebe8589feed3400aa6bca73df420778d P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 LEADER]: Becoming Leader. State: Replica: f215e8fdc8cb45f9a5eff7cb9e6ccfce, State: Running, Role: LEADER
I20240502 01:52:39.413854 27388 raft_consensus.cc:686] T fbd456fceb0f4c6db45662eb619cfd21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 LEADER]: Becoming Leader. State: Replica: f215e8fdc8cb45f9a5eff7cb9e6ccfce, State: Running, Role: LEADER
I20240502 01:52:39.413734 27503 consensus_queue.cc:237] T ebe8589feed3400aa6bca73df420778d P f215e8fdc8cb45f9a5eff7cb9e6ccfce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } }
I20240502 01:52:39.414168 27388 consensus_queue.cc:237] T fbd456fceb0f4c6db45662eb619cfd21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:39.417066 26077 catalog_manager.cc:5484] T fbd456fceb0f4c6db45662eb619cfd21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce reported cstate change: term changed from 0 to 1, leader changed from <none> to f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129). New cstate: current_term: 1 leader_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:39.417577 26077 catalog_manager.cc:5484] T ebe8589feed3400aa6bca73df420778d P f215e8fdc8cb45f9a5eff7cb9e6ccfce reported cstate change: term changed from 0 to 1, leader changed from <none> to f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129). New cstate: current_term: 1 leader_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } health_report { overall_health: UNKNOWN } } }
W20240502 01:52:39.794394 27511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:39.794728 27511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:39.802541 27511 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:52:39.814311 27475 consensus_queue.cc:1035] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:39.828399 27475 consensus_queue.cc:1035] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:39.848038 27503 consensus_queue.cc:1035] T ebe8589feed3400aa6bca73df420778d P f215e8fdc8cb45f9a5eff7cb9e6ccfce [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:39.853427 27503 consensus_queue.cc:1035] T ebe8589feed3400aa6bca73df420778d P f215e8fdc8cb45f9a5eff7cb9e6ccfce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
I20240502 01:52:39.892663 27388 consensus_queue.cc:1035] T fbd456fceb0f4c6db45662eb619cfd21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:39.901599 27508 consensus_queue.cc:1035] T fbd456fceb0f4c6db45662eb619cfd21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [LEADER]: Connected to new peer: Peer: permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 234.041 ms
  time per row: 0.117021 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      2        596 libcrypto.so.1.1
-----------------------------------------------------

I20240502 01:52:41.073623 26080 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57636:
name: "rebalance_test_table_4"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:52:41.083914 26335 tablet_service.cc:1467] Processing CreateTablet for tablet 062fa1dfab1046f287e2ddb1c240c4eb (DEFAULT_TABLE table=rebalance_test_table_4 [id=690e246e360e425598dedef6d62ce625]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:41.084482 26335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 062fa1dfab1046f287e2ddb1c240c4eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:41.084621 26601 tablet_service.cc:1467] Processing CreateTablet for tablet 062fa1dfab1046f287e2ddb1c240c4eb (DEFAULT_TABLE table=rebalance_test_table_4 [id=690e246e360e425598dedef6d62ce625]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:41.085081 26601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 062fa1dfab1046f287e2ddb1c240c4eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:41.085763 26599 tablet_service.cc:1467] Processing CreateTablet for tablet 7aa623407e7844989e7bf34db5d6ba35 (DEFAULT_TABLE table=rebalance_test_table_4 [id=690e246e360e425598dedef6d62ce625]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:41.086148 26599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aa623407e7844989e7bf34db5d6ba35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:41.091917 26336 tablet_service.cc:1467] Processing CreateTablet for tablet 64c74d4af3254bc28110d5b6d3b1c51f (DEFAULT_TABLE table=rebalance_test_table_4 [id=690e246e360e425598dedef6d62ce625]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:41.092399 26336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64c74d4af3254bc28110d5b6d3b1c51f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:41.093154 26467 tablet_service.cc:1467] Processing CreateTablet for tablet 64c74d4af3254bc28110d5b6d3b1c51f (DEFAULT_TABLE table=rebalance_test_table_4 [id=690e246e360e425598dedef6d62ce625]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:41.093605 26467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64c74d4af3254bc28110d5b6d3b1c51f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:41.093577 26202 tablet_service.cc:1467] Processing CreateTablet for tablet 64c74d4af3254bc28110d5b6d3b1c51f (DEFAULT_TABLE table=rebalance_test_table_4 [id=690e246e360e425598dedef6d62ce625]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:41.093575 26203 tablet_service.cc:1467] Processing CreateTablet for tablet 7aa623407e7844989e7bf34db5d6ba35 (DEFAULT_TABLE table=rebalance_test_table_4 [id=690e246e360e425598dedef6d62ce625]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:41.094030 26202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64c74d4af3254bc28110d5b6d3b1c51f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:41.094275 26203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aa623407e7844989e7bf34db5d6ba35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:41.095337 26469 tablet_service.cc:1467] Processing CreateTablet for tablet 7aa623407e7844989e7bf34db5d6ba35 (DEFAULT_TABLE table=rebalance_test_table_4 [id=690e246e360e425598dedef6d62ce625]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:41.095736 26469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aa623407e7844989e7bf34db5d6ba35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:41.097965 26204 tablet_service.cc:1467] Processing CreateTablet for tablet 062fa1dfab1046f287e2ddb1c240c4eb (DEFAULT_TABLE table=rebalance_test_table_4 [id=690e246e360e425598dedef6d62ce625]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:41.098412 26204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 062fa1dfab1046f287e2ddb1c240c4eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:41.112454 27582 tablet_bootstrap.cc:492] T 64c74d4af3254bc28110d5b6d3b1c51f P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Bootstrap starting.
I20240502 01:52:41.113634 27583 tablet_bootstrap.cc:492] T 062fa1dfab1046f287e2ddb1c240c4eb P 868e6e4358f4475bb45967b05eccb09f: Bootstrap starting.
I20240502 01:52:41.114110 27581 tablet_bootstrap.cc:492] T 7aa623407e7844989e7bf34db5d6ba35 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap starting.
I20240502 01:52:41.114709 27582 tablet_bootstrap.cc:654] T 64c74d4af3254bc28110d5b6d3b1c51f P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:41.115617 27581 tablet_bootstrap.cc:654] T 7aa623407e7844989e7bf34db5d6ba35 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:41.115753 27583 tablet_bootstrap.cc:654] T 062fa1dfab1046f287e2ddb1c240c4eb P 868e6e4358f4475bb45967b05eccb09f: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:41.117022 27580 tablet_bootstrap.cc:492] T 062fa1dfab1046f287e2ddb1c240c4eb P d1e7d0eabafa46f382e19f596e2fc00d: Bootstrap starting.
I20240502 01:52:41.117100 27582 tablet_bootstrap.cc:492] T 64c74d4af3254bc28110d5b6d3b1c51f P 6c70fbdc8a0b42ad8c09c6dbc842cc96: No bootstrap required, opened a new log
I20240502 01:52:41.117293 27582 ts_tablet_manager.cc:1367] T 64c74d4af3254bc28110d5b6d3b1c51f P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Time spent bootstrapping tablet: real 0.005s	user 0.002s	sys 0.002s
I20240502 01:52:41.118042 27581 tablet_bootstrap.cc:492] T 7aa623407e7844989e7bf34db5d6ba35 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: No bootstrap required, opened a new log
I20240502 01:52:41.118180 27581 ts_tablet_manager.cc:1367] T 7aa623407e7844989e7bf34db5d6ba35 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent bootstrapping tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 01:52:41.118098 27582 raft_consensus.cc:348] T 64c74d4af3254bc28110d5b6d3b1c51f P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:41.118359 27582 raft_consensus.cc:374] T 64c74d4af3254bc28110d5b6d3b1c51f P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:41.118465 27582 raft_consensus.cc:729] T 64c74d4af3254bc28110d5b6d3b1c51f P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c70fbdc8a0b42ad8c09c6dbc842cc96, State: Initialized, Role: FOLLOWER
I20240502 01:52:41.118659 27580 tablet_bootstrap.cc:654] T 062fa1dfab1046f287e2ddb1c240c4eb P d1e7d0eabafa46f382e19f596e2fc00d: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:41.118721 27582 consensus_queue.cc:260] T 64c74d4af3254bc28110d5b6d3b1c51f P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:41.118860 27581 raft_consensus.cc:348] T 7aa623407e7844989e7bf34db5d6ba35 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } }
I20240502 01:52:41.119064 27581 raft_consensus.cc:374] T 7aa623407e7844989e7bf34db5d6ba35 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:41.119156 27581 raft_consensus.cc:729] T 7aa623407e7844989e7bf34db5d6ba35 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f215e8fdc8cb45f9a5eff7cb9e6ccfce, State: Initialized, Role: FOLLOWER
I20240502 01:52:41.119405 27582 ts_tablet_manager.cc:1398] T 64c74d4af3254bc28110d5b6d3b1c51f P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 01:52:41.119364 27581 consensus_queue.cc:260] T 7aa623407e7844989e7bf34db5d6ba35 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } }
I20240502 01:52:41.120025 27581 ts_tablet_manager.cc:1398] T 7aa623407e7844989e7bf34db5d6ba35 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:52:41.119712 27582 tablet_bootstrap.cc:492] T 7aa623407e7844989e7bf34db5d6ba35 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Bootstrap starting.
I20240502 01:52:41.120380 27581 tablet_bootstrap.cc:492] T 64c74d4af3254bc28110d5b6d3b1c51f P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap starting.
I20240502 01:52:41.121115 27583 tablet_bootstrap.cc:492] T 062fa1dfab1046f287e2ddb1c240c4eb P 868e6e4358f4475bb45967b05eccb09f: No bootstrap required, opened a new log
I20240502 01:52:41.121327 27583 ts_tablet_manager.cc:1367] T 062fa1dfab1046f287e2ddb1c240c4eb P 868e6e4358f4475bb45967b05eccb09f: Time spent bootstrapping tablet: real 0.008s	user 0.003s	sys 0.000s
I20240502 01:52:41.122143 27583 raft_consensus.cc:348] T 062fa1dfab1046f287e2ddb1c240c4eb P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:52:41.122502 27583 raft_consensus.cc:374] T 062fa1dfab1046f287e2ddb1c240c4eb P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:41.122655 27583 raft_consensus.cc:729] T 062fa1dfab1046f287e2ddb1c240c4eb P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 868e6e4358f4475bb45967b05eccb09f, State: Initialized, Role: FOLLOWER
I20240502 01:52:41.122948 27583 consensus_queue.cc:260] T 062fa1dfab1046f287e2ddb1c240c4eb P 868e6e4358f4475bb45967b05eccb09f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:52:41.123530 27583 ts_tablet_manager.cc:1398] T 062fa1dfab1046f287e2ddb1c240c4eb P 868e6e4358f4475bb45967b05eccb09f: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 01:52:41.123901 27583 tablet_bootstrap.cc:492] T 64c74d4af3254bc28110d5b6d3b1c51f P 868e6e4358f4475bb45967b05eccb09f: Bootstrap starting.
I20240502 01:52:41.124508 27582 tablet_bootstrap.cc:654] T 7aa623407e7844989e7bf34db5d6ba35 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:41.125867 27583 tablet_bootstrap.cc:654] T 64c74d4af3254bc28110d5b6d3b1c51f P 868e6e4358f4475bb45967b05eccb09f: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:41.126614 27581 tablet_bootstrap.cc:654] T 64c74d4af3254bc28110d5b6d3b1c51f P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:41.128818 27582 tablet_bootstrap.cc:492] T 7aa623407e7844989e7bf34db5d6ba35 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: No bootstrap required, opened a new log
I20240502 01:52:41.128854 27580 tablet_bootstrap.cc:492] T 062fa1dfab1046f287e2ddb1c240c4eb P d1e7d0eabafa46f382e19f596e2fc00d: No bootstrap required, opened a new log
I20240502 01:52:41.129006 27580 ts_tablet_manager.cc:1367] T 062fa1dfab1046f287e2ddb1c240c4eb P d1e7d0eabafa46f382e19f596e2fc00d: Time spent bootstrapping tablet: real 0.012s	user 0.003s	sys 0.000s
I20240502 01:52:41.129103 27582 ts_tablet_manager.cc:1367] T 7aa623407e7844989e7bf34db5d6ba35 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Time spent bootstrapping tablet: real 0.009s	user 0.000s	sys 0.005s
I20240502 01:52:41.129781 27580 raft_consensus.cc:348] T 062fa1dfab1046f287e2ddb1c240c4eb P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:52:41.129859 27582 raft_consensus.cc:348] T 7aa623407e7844989e7bf34db5d6ba35 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } }
I20240502 01:52:41.130015 27580 raft_consensus.cc:374] T 062fa1dfab1046f287e2ddb1c240c4eb P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:41.130055 27581 tablet_bootstrap.cc:492] T 64c74d4af3254bc28110d5b6d3b1c51f P f215e8fdc8cb45f9a5eff7cb9e6ccfce: No bootstrap required, opened a new log
I20240502 01:52:41.130055 27582 raft_consensus.cc:374] T 7aa623407e7844989e7bf34db5d6ba35 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:41.130106 27580 raft_consensus.cc:729] T 062fa1dfab1046f287e2ddb1c240c4eb P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e7d0eabafa46f382e19f596e2fc00d, State: Initialized, Role: FOLLOWER
I20240502 01:52:41.130122 27582 raft_consensus.cc:729] T 7aa623407e7844989e7bf34db5d6ba35 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c70fbdc8a0b42ad8c09c6dbc842cc96, State: Initialized, Role: FOLLOWER
I20240502 01:52:41.130263 27581 ts_tablet_manager.cc:1367] T 64c74d4af3254bc28110d5b6d3b1c51f P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent bootstrapping tablet: real 0.010s	user 0.003s	sys 0.000s
I20240502 01:52:41.130359 27582 consensus_queue.cc:260] T 7aa623407e7844989e7bf34db5d6ba35 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } }
I20240502 01:52:41.130340 27580 consensus_queue.cc:260] T 062fa1dfab1046f287e2ddb1c240c4eb P d1e7d0eabafa46f382e19f596e2fc00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:52:41.130924 27580 ts_tablet_manager.cc:1398] T 062fa1dfab1046f287e2ddb1c240c4eb P d1e7d0eabafa46f382e19f596e2fc00d: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:52:41.130960 27581 raft_consensus.cc:348] T 64c74d4af3254bc28110d5b6d3b1c51f P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:41.131188 27581 raft_consensus.cc:374] T 64c74d4af3254bc28110d5b6d3b1c51f P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:41.131217 27580 tablet_bootstrap.cc:492] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d: Bootstrap starting.
I20240502 01:52:41.131263 27581 raft_consensus.cc:729] T 64c74d4af3254bc28110d5b6d3b1c51f P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f215e8fdc8cb45f9a5eff7cb9e6ccfce, State: Initialized, Role: FOLLOWER
I20240502 01:52:41.131546 27581 consensus_queue.cc:260] T 64c74d4af3254bc28110d5b6d3b1c51f P f215e8fdc8cb45f9a5eff7cb9e6ccfce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:41.132680 27581 ts_tablet_manager.cc:1398] T 64c74d4af3254bc28110d5b6d3b1c51f P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:52:41.132766 27580 tablet_bootstrap.cc:654] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:41.133000 27581 tablet_bootstrap.cc:492] T 062fa1dfab1046f287e2ddb1c240c4eb P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap starting.
I20240502 01:52:41.135017 27581 tablet_bootstrap.cc:654] T 062fa1dfab1046f287e2ddb1c240c4eb P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:41.135026 27583 tablet_bootstrap.cc:492] T 64c74d4af3254bc28110d5b6d3b1c51f P 868e6e4358f4475bb45967b05eccb09f: No bootstrap required, opened a new log
I20240502 01:52:41.135304 27583 ts_tablet_manager.cc:1367] T 64c74d4af3254bc28110d5b6d3b1c51f P 868e6e4358f4475bb45967b05eccb09f: Time spent bootstrapping tablet: real 0.011s	user 0.000s	sys 0.004s
I20240502 01:52:41.136148 27583 raft_consensus.cc:348] T 64c74d4af3254bc28110d5b6d3b1c51f P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:41.136440 27583 raft_consensus.cc:374] T 64c74d4af3254bc28110d5b6d3b1c51f P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:41.136559 27583 raft_consensus.cc:729] T 64c74d4af3254bc28110d5b6d3b1c51f P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 868e6e4358f4475bb45967b05eccb09f, State: Initialized, Role: FOLLOWER
I20240502 01:52:41.136780 27582 ts_tablet_manager.cc:1398] T 7aa623407e7844989e7bf34db5d6ba35 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Time spent starting tablet: real 0.008s	user 0.000s	sys 0.001s
I20240502 01:52:41.136873 27583 consensus_queue.cc:260] T 64c74d4af3254bc28110d5b6d3b1c51f P 868e6e4358f4475bb45967b05eccb09f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:41.137355 27581 tablet_bootstrap.cc:492] T 062fa1dfab1046f287e2ddb1c240c4eb P f215e8fdc8cb45f9a5eff7cb9e6ccfce: No bootstrap required, opened a new log
I20240502 01:52:41.137488 27581 ts_tablet_manager.cc:1367] T 062fa1dfab1046f287e2ddb1c240c4eb P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent bootstrapping tablet: real 0.005s	user 0.002s	sys 0.001s
I20240502 01:52:41.137568 27583 ts_tablet_manager.cc:1398] T 64c74d4af3254bc28110d5b6d3b1c51f P 868e6e4358f4475bb45967b05eccb09f: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:52:41.138159 27581 raft_consensus.cc:348] T 062fa1dfab1046f287e2ddb1c240c4eb P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:52:41.138366 27581 raft_consensus.cc:374] T 062fa1dfab1046f287e2ddb1c240c4eb P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:41.138469 27581 raft_consensus.cc:729] T 062fa1dfab1046f287e2ddb1c240c4eb P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f215e8fdc8cb45f9a5eff7cb9e6ccfce, State: Initialized, Role: FOLLOWER
I20240502 01:52:41.138701 27581 consensus_queue.cc:260] T 062fa1dfab1046f287e2ddb1c240c4eb P f215e8fdc8cb45f9a5eff7cb9e6ccfce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:52:41.139279 27580 tablet_bootstrap.cc:492] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d: No bootstrap required, opened a new log
I20240502 01:52:41.139442 27580 ts_tablet_manager.cc:1367] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d: Time spent bootstrapping tablet: real 0.008s	user 0.000s	sys 0.003s
I20240502 01:52:41.140058 27581 ts_tablet_manager.cc:1398] T 062fa1dfab1046f287e2ddb1c240c4eb P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 01:52:41.140170 27580 raft_consensus.cc:348] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } }
I20240502 01:52:41.140398 27580 raft_consensus.cc:374] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:41.140480 27580 raft_consensus.cc:729] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e7d0eabafa46f382e19f596e2fc00d, State: Initialized, Role: FOLLOWER
I20240502 01:52:41.140719 27580 consensus_queue.cc:260] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } }
I20240502 01:52:41.141310 27580 ts_tablet_manager.cc:1398] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240502 01:52:41.191210 27455 raft_consensus.cc:482] T 062fa1dfab1046f287e2ddb1c240c4eb P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:41.191421 27455 raft_consensus.cc:504] T 062fa1dfab1046f287e2ddb1c240c4eb P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:52:41.192406 27455 leader_election.cc:288] T 062fa1dfab1046f287e2ddb1c240c4eb P 868e6e4358f4475bb45967b05eccb09f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541), d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132:37695)
I20240502 01:52:41.192950 26223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "062fa1dfab1046f287e2ddb1c240c4eb" candidate_uuid: "868e6e4358f4475bb45967b05eccb09f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" is_pre_election: true
I20240502 01:52:41.193244 26223 raft_consensus.cc:2454] T 062fa1dfab1046f287e2ddb1c240c4eb P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 868e6e4358f4475bb45967b05eccb09f in term 0.
I20240502 01:52:41.193856 26292 leader_election.cc:302] T 062fa1dfab1046f287e2ddb1c240c4eb P 868e6e4358f4475bb45967b05eccb09f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 868e6e4358f4475bb45967b05eccb09f, f215e8fdc8cb45f9a5eff7cb9e6ccfce; no voters: 
I20240502 01:52:41.194190 27455 raft_consensus.cc:2781] T 062fa1dfab1046f287e2ddb1c240c4eb P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:41.194300 27455 raft_consensus.cc:482] T 062fa1dfab1046f287e2ddb1c240c4eb P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:41.194386 27455 raft_consensus.cc:3037] T 062fa1dfab1046f287e2ddb1c240c4eb P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:41.199570 26622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "062fa1dfab1046f287e2ddb1c240c4eb" candidate_uuid: "868e6e4358f4475bb45967b05eccb09f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" is_pre_election: true
I20240502 01:52:41.199836 26622 raft_consensus.cc:2454] T 062fa1dfab1046f287e2ddb1c240c4eb P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 868e6e4358f4475bb45967b05eccb09f in term 0.
I20240502 01:52:41.223917 27503 raft_consensus.cc:482] T 64c74d4af3254bc28110d5b6d3b1c51f P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:41.224165 27503 raft_consensus.cc:504] T 64c74d4af3254bc28110d5b6d3b1c51f P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:41.224898 27503 leader_election.cc:288] T 64c74d4af3254bc28110d5b6d3b1c51f P f215e8fdc8cb45f9a5eff7cb9e6ccfce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131:38693), 868e6e4358f4475bb45967b05eccb09f (127.25.109.130:45571)
I20240502 01:52:41.225296 26489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64c74d4af3254bc28110d5b6d3b1c51f" candidate_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" is_pre_election: true
I20240502 01:52:41.225292 26357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64c74d4af3254bc28110d5b6d3b1c51f" candidate_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "868e6e4358f4475bb45967b05eccb09f" is_pre_election: true
I20240502 01:52:41.225554 26489 raft_consensus.cc:2454] T 64c74d4af3254bc28110d5b6d3b1c51f P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f215e8fdc8cb45f9a5eff7cb9e6ccfce in term 0.
I20240502 01:52:41.225546 26357 raft_consensus.cc:2454] T 64c74d4af3254bc28110d5b6d3b1c51f P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f215e8fdc8cb45f9a5eff7cb9e6ccfce in term 0.
I20240502 01:52:41.225999 26158 leader_election.cc:302] T 64c74d4af3254bc28110d5b6d3b1c51f P f215e8fdc8cb45f9a5eff7cb9e6ccfce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6c70fbdc8a0b42ad8c09c6dbc842cc96, 868e6e4358f4475bb45967b05eccb09f, f215e8fdc8cb45f9a5eff7cb9e6ccfce; no voters: 
I20240502 01:52:41.226286 27503 raft_consensus.cc:2781] T 64c74d4af3254bc28110d5b6d3b1c51f P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:41.226379 27503 raft_consensus.cc:482] T 64c74d4af3254bc28110d5b6d3b1c51f P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:41.226482 27503 raft_consensus.cc:3037] T 64c74d4af3254bc28110d5b6d3b1c51f P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:41.237910 27233 raft_consensus.cc:482] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:41.238093 27233 raft_consensus.cc:504] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } }
I20240502 01:52:41.238871 27233 leader_election.cc:288] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131:38693), f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541)
I20240502 01:52:41.240303 26223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aa623407e7844989e7bf34db5d6ba35" candidate_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" is_pre_election: true
I20240502 01:52:41.240588 26223 raft_consensus.cc:2454] T 7aa623407e7844989e7bf34db5d6ba35 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e7d0eabafa46f382e19f596e2fc00d in term 0.
I20240502 01:52:41.241006 26556 leader_election.cc:302] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e7d0eabafa46f382e19f596e2fc00d, f215e8fdc8cb45f9a5eff7cb9e6ccfce; no voters: 
I20240502 01:52:41.241286 27233 raft_consensus.cc:2781] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:41.241395 27233 raft_consensus.cc:482] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:41.241479 27233 raft_consensus.cc:3037] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:41.245810 26489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aa623407e7844989e7bf34db5d6ba35" candidate_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" is_pre_election: true
I20240502 01:52:41.246042 26489 raft_consensus.cc:2454] T 7aa623407e7844989e7bf34db5d6ba35 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e7d0eabafa46f382e19f596e2fc00d in term 0.
I20240502 01:52:41.283340 27503 raft_consensus.cc:504] T 64c74d4af3254bc28110d5b6d3b1c51f P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:41.283376 27455 raft_consensus.cc:504] T 062fa1dfab1046f287e2ddb1c240c4eb P 868e6e4358f4475bb45967b05eccb09f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:52:41.284248 27503 leader_election.cc:288] T 64c74d4af3254bc28110d5b6d3b1c51f P f215e8fdc8cb45f9a5eff7cb9e6ccfce [CANDIDATE]: Term 1 election: Requested vote from peers 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131:38693), 868e6e4358f4475bb45967b05eccb09f (127.25.109.130:45571)
I20240502 01:52:41.284255 27455 leader_election.cc:288] T 062fa1dfab1046f287e2ddb1c240c4eb P 868e6e4358f4475bb45967b05eccb09f [CANDIDATE]: Term 1 election: Requested vote from peers f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541), d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132:37695)
I20240502 01:52:41.285647 27233 raft_consensus.cc:504] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } }
I20240502 01:52:41.285732 26357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64c74d4af3254bc28110d5b6d3b1c51f" candidate_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "868e6e4358f4475bb45967b05eccb09f"
I20240502 01:52:41.285985 26357 raft_consensus.cc:3037] T 64c74d4af3254bc28110d5b6d3b1c51f P 868e6e4358f4475bb45967b05eccb09f [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:41.286437 27233 leader_election.cc:288] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [CANDIDATE]: Term 1 election: Requested vote from peers 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131:38693), f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541)
I20240502 01:52:41.287428 26622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "062fa1dfab1046f287e2ddb1c240c4eb" candidate_uuid: "868e6e4358f4475bb45967b05eccb09f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1e7d0eabafa46f382e19f596e2fc00d"
I20240502 01:52:41.287681 26622 raft_consensus.cc:3037] T 062fa1dfab1046f287e2ddb1c240c4eb P d1e7d0eabafa46f382e19f596e2fc00d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:41.287943 26488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aa623407e7844989e7bf34db5d6ba35" candidate_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96"
I20240502 01:52:41.288188 26488 raft_consensus.cc:3037] T 7aa623407e7844989e7bf34db5d6ba35 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:41.289033 26223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "062fa1dfab1046f287e2ddb1c240c4eb" candidate_uuid: "868e6e4358f4475bb45967b05eccb09f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce"
I20240502 01:52:41.289276 26223 raft_consensus.cc:3037] T 062fa1dfab1046f287e2ddb1c240c4eb P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:41.290928 26489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64c74d4af3254bc28110d5b6d3b1c51f" candidate_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96"
I20240502 01:52:41.291178 26489 raft_consensus.cc:3037] T 64c74d4af3254bc28110d5b6d3b1c51f P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:41.291337 26222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aa623407e7844989e7bf34db5d6ba35" candidate_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce"
I20240502 01:52:41.291586 26222 raft_consensus.cc:3037] T 7aa623407e7844989e7bf34db5d6ba35 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:41.379261 26488 raft_consensus.cc:2454] T 7aa623407e7844989e7bf34db5d6ba35 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e7d0eabafa46f382e19f596e2fc00d in term 1.
I20240502 01:52:41.379261 26622 raft_consensus.cc:2454] T 062fa1dfab1046f287e2ddb1c240c4eb P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 868e6e4358f4475bb45967b05eccb09f in term 1.
I20240502 01:52:41.379272 26357 raft_consensus.cc:2454] T 64c74d4af3254bc28110d5b6d3b1c51f P 868e6e4358f4475bb45967b05eccb09f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f215e8fdc8cb45f9a5eff7cb9e6ccfce in term 1.
I20240502 01:52:41.379926 26291 leader_election.cc:302] T 062fa1dfab1046f287e2ddb1c240c4eb P 868e6e4358f4475bb45967b05eccb09f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 868e6e4358f4475bb45967b05eccb09f, d1e7d0eabafa46f382e19f596e2fc00d; no voters: 
I20240502 01:52:41.380164 26157 leader_election.cc:302] T 64c74d4af3254bc28110d5b6d3b1c51f P f215e8fdc8cb45f9a5eff7cb9e6ccfce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 868e6e4358f4475bb45967b05eccb09f, f215e8fdc8cb45f9a5eff7cb9e6ccfce; no voters: 
I20240502 01:52:41.380286 27455 raft_consensus.cc:2781] T 062fa1dfab1046f287e2ddb1c240c4eb P 868e6e4358f4475bb45967b05eccb09f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:41.380669 26223 raft_consensus.cc:2454] T 062fa1dfab1046f287e2ddb1c240c4eb P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 868e6e4358f4475bb45967b05eccb09f in term 1.
I20240502 01:52:41.380743 27503 raft_consensus.cc:2781] T 64c74d4af3254bc28110d5b6d3b1c51f P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:41.381598 26222 raft_consensus.cc:2454] T 7aa623407e7844989e7bf34db5d6ba35 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e7d0eabafa46f382e19f596e2fc00d in term 1.
I20240502 01:52:41.382275 27503 raft_consensus.cc:686] T 64c74d4af3254bc28110d5b6d3b1c51f P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 LEADER]: Becoming Leader. State: Replica: f215e8fdc8cb45f9a5eff7cb9e6ccfce, State: Running, Role: LEADER
I20240502 01:52:41.382313 26489 raft_consensus.cc:2454] T 64c74d4af3254bc28110d5b6d3b1c51f P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f215e8fdc8cb45f9a5eff7cb9e6ccfce in term 1.
I20240502 01:52:41.382625 26556 leader_election.cc:302] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1e7d0eabafa46f382e19f596e2fc00d, f215e8fdc8cb45f9a5eff7cb9e6ccfce; no voters: 
I20240502 01:52:41.382956 27233 raft_consensus.cc:2781] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:41.382810 27503 consensus_queue.cc:237] T 64c74d4af3254bc28110d5b6d3b1c51f P f215e8fdc8cb45f9a5eff7cb9e6ccfce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:41.383121 27233 raft_consensus.cc:686] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 LEADER]: Becoming Leader. State: Replica: d1e7d0eabafa46f382e19f596e2fc00d, State: Running, Role: LEADER
I20240502 01:52:41.383423 27233 consensus_queue.cc:237] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } }
I20240502 01:52:41.384487 27455 raft_consensus.cc:686] T 062fa1dfab1046f287e2ddb1c240c4eb P 868e6e4358f4475bb45967b05eccb09f [term 1 LEADER]: Becoming Leader. State: Replica: 868e6e4358f4475bb45967b05eccb09f, State: Running, Role: LEADER
I20240502 01:52:41.384846 27455 consensus_queue.cc:237] T 062fa1dfab1046f287e2ddb1c240c4eb P 868e6e4358f4475bb45967b05eccb09f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:52:41.387632 26075 catalog_manager.cc:5484] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d reported cstate change: term changed from 0 to 1, leader changed from <none> to d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132). New cstate: current_term: 1 leader_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:41.387595 26080 catalog_manager.cc:5484] T 64c74d4af3254bc28110d5b6d3b1c51f P f215e8fdc8cb45f9a5eff7cb9e6ccfce reported cstate change: term changed from 0 to 1, leader changed from <none> to f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129). New cstate: current_term: 1 leader_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:41.388392 26077 catalog_manager.cc:5484] T 062fa1dfab1046f287e2ddb1c240c4eb P 868e6e4358f4475bb45967b05eccb09f reported cstate change: term changed from 0 to 1, leader changed from <none> to 868e6e4358f4475bb45967b05eccb09f (127.25.109.130). New cstate: current_term: 1 leader_uuid: "868e6e4358f4475bb45967b05eccb09f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:41.794199 27455 consensus_queue.cc:1035] T 062fa1dfab1046f287e2ddb1c240c4eb P 868e6e4358f4475bb45967b05eccb09f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:41.800387 27599 consensus_queue.cc:1035] T 062fa1dfab1046f287e2ddb1c240c4eb P 868e6e4358f4475bb45967b05eccb09f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:41.814846 27600 consensus_queue.cc:1035] T 64c74d4af3254bc28110d5b6d3b1c51f P f215e8fdc8cb45f9a5eff7cb9e6ccfce [LEADER]: Connected to new peer: Peer: permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:52:41.816967 27605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:41.817483 27605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:41.829731 27605 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:52:41.832643 27503 consensus_queue.cc:1035] T 64c74d4af3254bc28110d5b6d3b1c51f P f215e8fdc8cb45f9a5eff7cb9e6ccfce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:41.836612 27233 consensus_queue.cc:1035] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:41.846401 27598 consensus_queue.cc:1035] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_4'
W20240502 01:52:42.153481 26291 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.132:37695, user_credentials={real_user=slave}} blocked reactor thread for 65277.2us

INSERT report
    rows total: 2000
    time total: 257.429 ms
  time per row: 0.128714 ms
I20240502 01:52:42.254638 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.133:33081
--local_ip_for_outbound_sockets=127.25.109.133
--tserver_master_addrs=127.25.109.190:44417
--webserver_port=38707
--webserver_interface=127.25.109.133
--builtin_ntp_servers=127.25.109.148:34611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240502 01:52:42.614413 27671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:42.614775 27671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:42.614946 27671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:42.623306 27671 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:42.623452 27671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:42.623641 27671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.133
I20240502 01:52:42.632937 27671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:34611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.133:33081
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.25.109.133
--webserver_port=38707
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.25.109.190:44417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.133
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:42.634430 27671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:42.636456 27671 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:42.649619 27679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:42.649988 27677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:42.650735 27676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:42.651252 27671 server_base.cc:1034] running on GCE node
I20240502 01:52:42.651998 27671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:42.653373 27671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:42.654700 27671 hybrid_clock.cc:648] HybridClock initialized: now 1714614762654640 us; error 58 us; skew 500 ppm
I20240502 01:52:42.655078 27671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:42.657647 27671 webserver.cc:460] Webserver started at http://127.25.109.133:38707/ using document root <none> and password file <none>
I20240502 01:52:42.658463 27671 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:42.658597 27671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:42.664572 27671 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.002s	sys 0.001s
I20240502 01:52:42.668056 27685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:42.669636 27671 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240502 01:52:42.669797 27671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "ca931e9360d44079b6059b8d9e7a6d20"
format_stamp: "Formatted at 2024-05-02 01:52:29 on dist-test-slave-kj4b"
I20240502 01:52:42.670727 27671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:42.701894 27671 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:52:42.702757 27671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:42.703032 27671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:42.704073 27671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:42.705747 27671 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:42.705878 27671 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.001s	sys 0.000s
I20240502 01:52:42.706015 27671 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:42.706086 27671 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:42.742167 27671 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.133:33081
I20240502 01:52:42.742200 27801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.133:33081 every 8 connection(s)
I20240502 01:52:42.743697 27671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:52:42.753863 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 27671
I20240502 01:52:42.754535 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.134:43811
--local_ip_for_outbound_sockets=127.25.109.134
--tserver_master_addrs=127.25.109.190:44417
--webserver_port=44899
--webserver_interface=127.25.109.134
--builtin_ntp_servers=127.25.109.148:34611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 01:52:42.757946 27802 heartbeater.cc:347] Connected to a master server at 127.25.109.190:44417
I20240502 01:52:42.758282 27802 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:42.759212 27802 heartbeater.cc:511] Master 127.25.109.190:44417 requested a full tablet report, sending...
I20240502 01:52:42.760951 26075 ts_manager.cc:194] Re-registered known tserver with Master: ca931e9360d44079b6059b8d9e7a6d20 (127.25.109.133:33081)
I20240502 01:52:42.761865 26075 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.133:40399
W20240502 01:52:43.097959 27807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:43.098412 27807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:43.098582 27807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:43.106215 27807 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:43.106343 27807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:43.106462 27807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.134
I20240502 01:52:43.116137 27807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:34611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.134:43811
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.25.109.134
--webserver_port=44899
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.25.109.190:44417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.134
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:43.117645 27807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:43.119486 27807 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:43.131798 27813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:43.132153 27812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:43.132719 27815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:43.133419 27807 server_base.cc:1034] running on GCE node
I20240502 01:52:43.134042 27807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:43.135071 27807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:43.136265 27807 hybrid_clock.cc:648] HybridClock initialized: now 1714614763136201 us; error 65 us; skew 500 ppm
I20240502 01:52:43.136587 27807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:43.139272 27807 webserver.cc:460] Webserver started at http://127.25.109.134:44899/ using document root <none> and password file <none>
I20240502 01:52:43.140095 27807 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:43.140228 27807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:43.145493 27807 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.001s	sys 0.004s
I20240502 01:52:43.148463 27821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:43.149869 27807 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20240502 01:52:43.150035 27807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "6ecf0a12e15f45a3bfa042f88c100a5e"
format_stamp: "Formatted at 2024-05-02 01:52:29 on dist-test-slave-kj4b"
I20240502 01:52:43.150934 27807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:43.191398 27807 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:52:43.192354 27807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:43.192641 27807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:43.193626 27807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:43.195183 27807 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:43.195259 27807 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:43.195386 27807 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:43.195437 27807 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:43.225317 27807 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.134:43811
I20240502 01:52:43.225376 27936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.134:43811 every 8 connection(s)
I20240502 01:52:43.226845 27807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 01:52:43.233273 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 27807
I20240502 01:52:43.233866 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.135:39905
--local_ip_for_outbound_sockets=127.25.109.135
--tserver_master_addrs=127.25.109.190:44417
--webserver_port=39507
--webserver_interface=127.25.109.135
--builtin_ntp_servers=127.25.109.148:34611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 01:52:43.240314 27937 heartbeater.cc:347] Connected to a master server at 127.25.109.190:44417
I20240502 01:52:43.240687 27937 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:43.241737 27937 heartbeater.cc:511] Master 127.25.109.190:44417 requested a full tablet report, sending...
I20240502 01:52:43.243263 26075 ts_manager.cc:194] Re-registered known tserver with Master: 6ecf0a12e15f45a3bfa042f88c100a5e (127.25.109.134:43811)
I20240502 01:52:43.244199 26075 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.134:59071
W20240502 01:52:43.643101 27940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:43.643465 27940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:43.643610 27940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:43.651995 27940 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:43.652211 27940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:43.652419 27940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.135
I20240502 01:52:43.663100 27940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:34611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.135:39905
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.25.109.135
--webserver_port=39507
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.25.109.190:44417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.135
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:43.664578 27940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:43.666450 27940 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:43.680423 27948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:43.681062 27947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:43.681577 27950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:43.682132 27940 server_base.cc:1034] running on GCE node
I20240502 01:52:43.682909 27940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:43.684108 27940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:43.685326 27940 hybrid_clock.cc:648] HybridClock initialized: now 1714614763685258 us; error 61 us; skew 500 ppm
I20240502 01:52:43.685684 27940 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:43.688412 27940 webserver.cc:460] Webserver started at http://127.25.109.135:39507/ using document root <none> and password file <none>
I20240502 01:52:43.689251 27940 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:43.689390 27940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:43.695734 27940 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.006s	sys 0.000s
I20240502 01:52:43.698992 27956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:43.700466 27940 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 01:52:43.700642 27940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "aed397efb8844970b606bca15a77cb59"
format_stamp: "Formatted at 2024-05-02 01:52:30 on dist-test-slave-kj4b"
I20240502 01:52:43.701786 27940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:43.724014 27940 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:52:43.725229 27940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:43.725713 27940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:43.726856 27940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:43.728757 27940 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:43.728859 27940 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:43.728981 27940 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:43.729039 27940 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:43.764636 27802 heartbeater.cc:503] Master 127.25.109.190:44417 was elected leader, sending a full tablet report...
I20240502 01:52:43.783218 27940 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.135:39905
I20240502 01:52:43.783273 28070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.135:39905 every 8 connection(s)
I20240502 01:52:43.784816 27940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 01:52:43.794523 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 27940
I20240502 01:52:43.795320 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.136:38319
--local_ip_for_outbound_sockets=127.25.109.136
--tserver_master_addrs=127.25.109.190:44417
--webserver_port=33819
--webserver_interface=127.25.109.136
--builtin_ntp_servers=127.25.109.148:34611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 01:52:43.800531 28071 heartbeater.cc:347] Connected to a master server at 127.25.109.190:44417
I20240502 01:52:43.800892 28071 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:43.802242 28071 heartbeater.cc:511] Master 127.25.109.190:44417 requested a full tablet report, sending...
I20240502 01:52:43.804067 26075 ts_manager.cc:194] Re-registered known tserver with Master: aed397efb8844970b606bca15a77cb59 (127.25.109.135:39905)
I20240502 01:52:43.804876 26075 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.135:58677
W20240502 01:52:44.184285 28075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:44.184619 28075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:44.184772 28075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:44.193003 28075 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:44.193166 28075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:44.193326 28075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.136
I20240502 01:52:44.203806 28075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:34611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.136:38319
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.25.109.136
--webserver_port=33819
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.25.109.190:44417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.136
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:44.205297 28075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:44.207312 28075 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:44.220091 28080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:44.220683 28081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:44.221060 28083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:44.225764 28075 server_base.cc:1034] running on GCE node
I20240502 01:52:44.226583 28075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:44.227975 28075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:44.229220 28075 hybrid_clock.cc:648] HybridClock initialized: now 1714614764229153 us; error 54 us; skew 500 ppm
I20240502 01:52:44.229624 28075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:44.232090 28075 webserver.cc:460] Webserver started at http://127.25.109.136:33819/ using document root <none> and password file <none>
I20240502 01:52:44.232949 28075 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:44.233105 28075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:44.238654 28075 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.004s	sys 0.002s
I20240502 01:52:44.241564 28089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:44.242938 28075 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20240502 01:52:44.243104 28075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "c292d733f9584891b46f52aafee403fc"
format_stamp: "Formatted at 2024-05-02 01:52:30 on dist-test-slave-kj4b"
I20240502 01:52:44.244012 28075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:44.247454 27937 heartbeater.cc:503] Master 127.25.109.190:44417 was elected leader, sending a full tablet report...
I20240502 01:52:44.267544 28075 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:52:44.268714 28075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:44.269168 28075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:44.270329 28075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:44.272296 28075 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:44.272382 28075 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:44.272514 28075 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:44.272565 28075 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:44.307252 28075 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.136:38319
I20240502 01:52:44.307740 28205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.136:38319 every 8 connection(s)
I20240502 01:52:44.308863 28075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 01:52:44.315526 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 28075
I20240502 01:52:44.322959 28206 heartbeater.cc:347] Connected to a master server at 127.25.109.190:44417
I20240502 01:52:44.323316 28206 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:44.324271 28206 heartbeater.cc:511] Master 127.25.109.190:44417 requested a full tablet report, sending...
I20240502 01:52:44.325783 26075 ts_manager.cc:194] Re-registered known tserver with Master: c292d733f9584891b46f52aafee403fc (127.25.109.136:38319)
I20240502 01:52:44.326625 26075 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.136:54283
I20240502 01:52:44.808492 28071 heartbeater.cc:503] Master 127.25.109.190:44417 was elected leader, sending a full tablet report...
I20240502 01:52:44.865666 26467 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:52:44.875577 27869 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:44.881491 26335 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:52:44.902840 28139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:44.907570 26601 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:52:44.916890 28005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:44.919011 27736 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:44.933084 26203 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:52:45.329311 28206 heartbeater.cc:503] Master 127.25.109.190:44417 was elected leader, sending a full tablet report...
I20240502 01:52:45.503657 26075 ts_manager.cc:292] Set tserver state for f215e8fdc8cb45f9a5eff7cb9e6ccfce to MAINTENANCE_MODE
W20240502 01:52:46.840411 28277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:46.840762 28277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:46.848698 28277 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:52:47.044858 26467 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:52:47.053700 27869 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:47.065414 26335 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:52:47.102142 26467 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:52:47.140435 27736 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:47.145462 27869 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:47.149919 26335 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:52:47.169314 26203 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:52:47.173588 28139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:47.207352 28005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:47.219800 26601 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:52:47.259671 27736 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:47.276433 28005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:47.279111 28139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:47.307456 26601 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:52:47.319435 26203 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:52:47.535293 26467 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:52:47.575106 27869 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:47.576411 27736 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:47.586656 26601 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:52:47.596475 26203 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:52:47.607831 26335 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:52:47.620546 28139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:47.623845 28005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:47.883152 27869 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:47.884275 26467 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:52:47.924757 28139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:47.944806 26335 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:52:47.977536 28005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:48.015743 26601 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:52:48.047974 27736 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:48.071851 26203 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:52:48.138924 28405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:48.139524 28405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:48.151283 28405 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:52:48.170157 26621 consensus_queue.cc:237] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } attrs { replace: true } }
I20240502 01:52:48.173290 26223 raft_consensus.cc:1261] T 7aa623407e7844989e7bf34db5d6ba35 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Refusing update from remote peer d1e7d0eabafa46f382e19f596e2fc00d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:52:48.173339 26488 raft_consensus.cc:1261] T 7aa623407e7844989e7bf34db5d6ba35 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Refusing update from remote peer d1e7d0eabafa46f382e19f596e2fc00d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:52:48.174136 28458 consensus_queue.cc:1035] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:52:48.174567 27650 consensus_queue.cc:1035] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:52:48.178467 27650 raft_consensus.cc:2932] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } attrs { replace: true } } }
I20240502 01:52:48.179378 26223 raft_consensus.cc:2932] T 7aa623407e7844989e7bf34db5d6ba35 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } attrs { replace: true } } }
I20240502 01:52:48.180027 26488 raft_consensus.cc:2932] T 7aa623407e7844989e7bf34db5d6ba35 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } attrs { replace: true } } }
I20240502 01:52:48.186486 26076 catalog_manager.cc:5484] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } attrs { replace: true } health_report { overall_health: HEALTHY } } }
I20240502 01:52:48.206195 26621 consensus_queue.cc:237] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: NON_VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: true } }
I20240502 01:52:48.209306 26223 raft_consensus.cc:1261] T 7aa623407e7844989e7bf34db5d6ba35 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Refusing update from remote peer d1e7d0eabafa46f382e19f596e2fc00d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:52:48.209559 26490 raft_consensus.cc:1261] T 7aa623407e7844989e7bf34db5d6ba35 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Refusing update from remote peer d1e7d0eabafa46f382e19f596e2fc00d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:52:48.216187 28462 consensus_queue.cc:1035] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:52:48.216879 28462 consensus_queue.cc:1035] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:52:48.218922 28458 raft_consensus.cc:2932] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c292d733f9584891b46f52aafee403fc (127.25.109.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: NON_VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: true } } }
I20240502 01:52:48.220114 26490 raft_consensus.cc:2932] T 7aa623407e7844989e7bf34db5d6ba35 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c292d733f9584891b46f52aafee403fc (127.25.109.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: NON_VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: true } } }
I20240502 01:52:48.222841 26067 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 7aa623407e7844989e7bf34db5d6ba35 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 01:52:48.224305 26075 catalog_manager.cc:5484] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d reported cstate change: config changed from index 4 to 5, NON_VOTER c292d733f9584891b46f52aafee403fc (127.25.109.136) added. New cstate: current_term: 1 leader_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: NON_VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:48.227305 26223 raft_consensus.cc:2932] T 7aa623407e7844989e7bf34db5d6ba35 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c292d733f9584891b46f52aafee403fc (127.25.109.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: NON_VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: true } } }
W20240502 01:52:48.246141 26554 consensus_peers.cc:485] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer c292d733f9584891b46f52aafee403fc (127.25.109.136:38319): Couldn't send request to peer c292d733f9584891b46f52aafee403fc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7aa623407e7844989e7bf34db5d6ba35. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:52:48.330094 27869 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:48.330880 26467 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:52:48.336776 28005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:48.353229 28139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:48.354749 26335 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:52:48.358884 27736 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:48.396533 26601 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:52:48.411151 26203 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:52:48.743651 28513 ts_tablet_manager.cc:902] T 7aa623407e7844989e7bf34db5d6ba35 P c292d733f9584891b46f52aafee403fc: Initiating tablet copy from peer d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132:37695)
I20240502 01:52:48.749034 28513 tablet_copy_client.cc:308] T 7aa623407e7844989e7bf34db5d6ba35 P c292d733f9584891b46f52aafee403fc: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.132:37695
I20240502 01:52:48.784482 26642 tablet_copy_service.cc:140] P d1e7d0eabafa46f382e19f596e2fc00d: Received BeginTabletCopySession request for tablet 7aa623407e7844989e7bf34db5d6ba35 from peer c292d733f9584891b46f52aafee403fc ({username='slave'} at 127.25.109.136:48021)
I20240502 01:52:48.784795 26642 tablet_copy_service.cc:161] P d1e7d0eabafa46f382e19f596e2fc00d: Beginning new tablet copy session on tablet 7aa623407e7844989e7bf34db5d6ba35 from peer c292d733f9584891b46f52aafee403fc at {username='slave'} at 127.25.109.136:48021: session id = c292d733f9584891b46f52aafee403fc-7aa623407e7844989e7bf34db5d6ba35
I20240502 01:52:48.788750 26642 tablet_copy_source_session.cc:205] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:52:48.795346 28513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aa623407e7844989e7bf34db5d6ba35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:48.806147 28513 tablet_copy_client.cc:783] T 7aa623407e7844989e7bf34db5d6ba35 P c292d733f9584891b46f52aafee403fc: tablet copy: Starting download of 0 data blocks...
I20240502 01:52:48.806656 28513 tablet_copy_client.cc:647] T 7aa623407e7844989e7bf34db5d6ba35 P c292d733f9584891b46f52aafee403fc: tablet copy: Starting download of 1 WAL segments...
I20240502 01:52:48.809243 28517 tablet_copy_client.cc:1002] T 7aa623407e7844989e7bf34db5d6ba35 P c292d733f9584891b46f52aafee403fc: tablet copy: Injecting latency into file download: 1500
W20240502 01:52:48.943723 28507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:48.944087 28507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:48.951848 28507 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:52:49.105453 26335 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:52:49.115049 27869 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:49.125581 26601 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:52:49.163487 26467 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:52:49.183898 26203 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:52:49.184839 28139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:49.186407 28005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:49.186808 27736 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Runtime error: ksck discovered errors
I20240502 01:52:49.380748 28275 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 26140
I20240502 01:52:49.412137 26264 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:52:49.542294 28275 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 26140
W20240502 01:52:49.562391 26555 connection.cc:535] server connection from 127.25.109.129:56463 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:52:49.622165 26292 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111)
W20240502 01:52:49.623579 26292 consensus_peers.cc:485] T 7def725255ba4448837b47ba65a6ed44 P 868e6e4358f4475bb45967b05eccb09f -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:52:49.655761 26556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111)
W20240502 01:52:49.657477 26556 consensus_peers.cc:485] T 01b7ac587d134a61b333489d937c0a21 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:52:49.712955 26292 consensus_peers.cc:485] T c32bdf0d644e499eaa2f755fab9abb77 P 868e6e4358f4475bb45967b05eccb09f -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:52:49.844458 26292 consensus_peers.cc:485] T 062fa1dfab1046f287e2ddb1c240c4eb P 868e6e4358f4475bb45967b05eccb09f -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:52:49.877089 26556 consensus_peers.cc:485] T dc44aad1f4ab4469a29d280a11268be0 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:52:49.902137 26556 consensus_peers.cc:485] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:52:49.989437 26556 consensus_peers.cc:485] T 536039b70452486684f7075cf45c1851 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:52:50.309782 28513 tablet_copy_client.cc:523] T 7aa623407e7844989e7bf34db5d6ba35 P c292d733f9584891b46f52aafee403fc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:52:50.313305 28513 tablet_bootstrap.cc:492] T 7aa623407e7844989e7bf34db5d6ba35 P c292d733f9584891b46f52aafee403fc: Bootstrap starting.
I20240502 01:52:50.322743 28513 log.cc:824] T 7aa623407e7844989e7bf34db5d6ba35 P c292d733f9584891b46f52aafee403fc: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:50.356084 28513 tablet_bootstrap.cc:492] T 7aa623407e7844989e7bf34db5d6ba35 P c292d733f9584891b46f52aafee403fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:52:50.356628 28513 tablet_bootstrap.cc:492] T 7aa623407e7844989e7bf34db5d6ba35 P c292d733f9584891b46f52aafee403fc: Bootstrap complete.
I20240502 01:52:50.357293 28513 ts_tablet_manager.cc:1367] T 7aa623407e7844989e7bf34db5d6ba35 P c292d733f9584891b46f52aafee403fc: Time spent bootstrapping tablet: real 0.044s	user 0.039s	sys 0.006s
I20240502 01:52:50.362241 28513 raft_consensus.cc:348] T 7aa623407e7844989e7bf34db5d6ba35 P c292d733f9584891b46f52aafee403fc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: NON_VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: true } }
I20240502 01:52:50.362800 28513 raft_consensus.cc:729] T 7aa623407e7844989e7bf34db5d6ba35 P c292d733f9584891b46f52aafee403fc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c292d733f9584891b46f52aafee403fc, State: Initialized, Role: LEARNER
I20240502 01:52:50.363360 28513 consensus_queue.cc:260] T 7aa623407e7844989e7bf34db5d6ba35 P c292d733f9584891b46f52aafee403fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: NON_VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: true } }
I20240502 01:52:50.365056 28513 ts_tablet_manager.cc:1398] T 7aa623407e7844989e7bf34db5d6ba35 P c292d733f9584891b46f52aafee403fc: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20240502 01:52:50.366124 26642 tablet_copy_service.cc:342] P d1e7d0eabafa46f382e19f596e2fc00d: Request end of tablet copy session c292d733f9584891b46f52aafee403fc-7aa623407e7844989e7bf34db5d6ba35 received from {username='slave'} at 127.25.109.136:48021
I20240502 01:52:50.366333 26642 tablet_copy_service.cc:433] P d1e7d0eabafa46f382e19f596e2fc00d: ending tablet copy session c292d733f9584891b46f52aafee403fc-7aa623407e7844989e7bf34db5d6ba35 on tablet 7aa623407e7844989e7bf34db5d6ba35 with peer c292d733f9584891b46f52aafee403fc
I20240502 01:52:50.663295 28442 raft_consensus.cc:482] T 4e6854ffbc944e45b51256143505cdb8 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f215e8fdc8cb45f9a5eff7cb9e6ccfce)
I20240502 01:52:50.663511 28442 raft_consensus.cc:504] T 4e6854ffbc944e45b51256143505cdb8 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:52:50.664557 28442 leader_election.cc:288] T 4e6854ffbc944e45b51256143505cdb8 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541), d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132:37695)
I20240502 01:52:50.664948 28160 raft_consensus.cc:1203] T 7aa623407e7844989e7bf34db5d6ba35 P c292d733f9584891b46f52aafee403fc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 01:52:50.665148 26621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e6854ffbc944e45b51256143505cdb8" candidate_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" is_pre_election: true
W20240502 01:52:50.665685 26424 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111)
W20240502 01:52:50.667217 26424 leader_election.cc:334] T 4e6854ffbc944e45b51256143505cdb8 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111)
I20240502 01:52:50.667376 26424 leader_election.cc:302] T 4e6854ffbc944e45b51256143505cdb8 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c70fbdc8a0b42ad8c09c6dbc842cc96; no voters: d1e7d0eabafa46f382e19f596e2fc00d, f215e8fdc8cb45f9a5eff7cb9e6ccfce
I20240502 01:52:50.667675 28442 raft_consensus.cc:2726] T 4e6854ffbc944e45b51256143505cdb8 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:52:50.728443 28442 raft_consensus.cc:482] T fbd456fceb0f4c6db45662eb619cfd21 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f215e8fdc8cb45f9a5eff7cb9e6ccfce)
I20240502 01:52:50.728652 28442 raft_consensus.cc:504] T fbd456fceb0f4c6db45662eb619cfd21 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:50.729524 28442 leader_election.cc:288] T fbd456fceb0f4c6db45662eb619cfd21 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541), 868e6e4358f4475bb45967b05eccb09f (127.25.109.130:45571)
I20240502 01:52:50.730201 26357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbd456fceb0f4c6db45662eb619cfd21" candidate_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "868e6e4358f4475bb45967b05eccb09f" is_pre_election: true
W20240502 01:52:50.731069 26424 leader_election.cc:334] T fbd456fceb0f4c6db45662eb619cfd21 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111)
I20240502 01:52:50.731230 26424 leader_election.cc:302] T fbd456fceb0f4c6db45662eb619cfd21 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c70fbdc8a0b42ad8c09c6dbc842cc96; no voters: 868e6e4358f4475bb45967b05eccb09f, f215e8fdc8cb45f9a5eff7cb9e6ccfce
I20240502 01:52:50.731489 28442 raft_consensus.cc:2726] T fbd456fceb0f4c6db45662eb619cfd21 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:52:50.768642 28610 raft_consensus.cc:482] T fbd456fceb0f4c6db45662eb619cfd21 P 868e6e4358f4475bb45967b05eccb09f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f215e8fdc8cb45f9a5eff7cb9e6ccfce)
I20240502 01:52:50.768880 28610 raft_consensus.cc:504] T fbd456fceb0f4c6db45662eb619cfd21 P 868e6e4358f4475bb45967b05eccb09f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:50.769793 28610 leader_election.cc:288] T fbd456fceb0f4c6db45662eb619cfd21 P 868e6e4358f4475bb45967b05eccb09f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131:38693), f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541)
I20240502 01:52:50.770637 26490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbd456fceb0f4c6db45662eb619cfd21" candidate_uuid: "868e6e4358f4475bb45967b05eccb09f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" is_pre_election: true
I20240502 01:52:50.770958 26490 raft_consensus.cc:2454] T fbd456fceb0f4c6db45662eb619cfd21 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 868e6e4358f4475bb45967b05eccb09f in term 1.
I20240502 01:52:50.771829 26292 leader_election.cc:302] T fbd456fceb0f4c6db45662eb619cfd21 P 868e6e4358f4475bb45967b05eccb09f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c70fbdc8a0b42ad8c09c6dbc842cc96, 868e6e4358f4475bb45967b05eccb09f; no voters: 
I20240502 01:52:50.772153 28610 raft_consensus.cc:2781] T fbd456fceb0f4c6db45662eb619cfd21 P 868e6e4358f4475bb45967b05eccb09f [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240502 01:52:50.772176 26292 leader_election.cc:334] T fbd456fceb0f4c6db45662eb619cfd21 P 868e6e4358f4475bb45967b05eccb09f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111)
I20240502 01:52:50.772306 28610 raft_consensus.cc:482] T fbd456fceb0f4c6db45662eb619cfd21 P 868e6e4358f4475bb45967b05eccb09f [term 1 FOLLOWER]: Starting leader election (detected failure of leader f215e8fdc8cb45f9a5eff7cb9e6ccfce)
I20240502 01:52:50.772403 28610 raft_consensus.cc:3037] T fbd456fceb0f4c6db45662eb619cfd21 P 868e6e4358f4475bb45967b05eccb09f [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:50.775050 28610 raft_consensus.cc:504] T fbd456fceb0f4c6db45662eb619cfd21 P 868e6e4358f4475bb45967b05eccb09f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:50.775975 28610 leader_election.cc:288] T fbd456fceb0f4c6db45662eb619cfd21 P 868e6e4358f4475bb45967b05eccb09f [CANDIDATE]: Term 2 election: Requested vote from peers 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131:38693), f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541)
I20240502 01:52:50.776566 26490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbd456fceb0f4c6db45662eb619cfd21" candidate_uuid: "868e6e4358f4475bb45967b05eccb09f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96"
I20240502 01:52:50.776799 26490 raft_consensus.cc:3037] T fbd456fceb0f4c6db45662eb619cfd21 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:52:50.777578 26292 leader_election.cc:334] T fbd456fceb0f4c6db45662eb619cfd21 P 868e6e4358f4475bb45967b05eccb09f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111)
I20240502 01:52:50.779156 26490 raft_consensus.cc:2454] T fbd456fceb0f4c6db45662eb619cfd21 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 868e6e4358f4475bb45967b05eccb09f in term 2.
I20240502 01:52:50.779611 26292 leader_election.cc:302] T fbd456fceb0f4c6db45662eb619cfd21 P 868e6e4358f4475bb45967b05eccb09f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6c70fbdc8a0b42ad8c09c6dbc842cc96, 868e6e4358f4475bb45967b05eccb09f; no voters: f215e8fdc8cb45f9a5eff7cb9e6ccfce
I20240502 01:52:50.779961 28610 raft_consensus.cc:2781] T fbd456fceb0f4c6db45662eb619cfd21 P 868e6e4358f4475bb45967b05eccb09f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:52:50.779999 28604 raft_consensus.cc:482] T 4e6854ffbc944e45b51256143505cdb8 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f215e8fdc8cb45f9a5eff7cb9e6ccfce)
I20240502 01:52:50.780128 28610 raft_consensus.cc:686] T fbd456fceb0f4c6db45662eb619cfd21 P 868e6e4358f4475bb45967b05eccb09f [term 2 LEADER]: Becoming Leader. State: Replica: 868e6e4358f4475bb45967b05eccb09f, State: Running, Role: LEADER
I20240502 01:52:50.780177 28604 raft_consensus.cc:504] T 4e6854ffbc944e45b51256143505cdb8 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:52:50.780438 28610 consensus_queue.cc:237] T fbd456fceb0f4c6db45662eb619cfd21 P 868e6e4358f4475bb45967b05eccb09f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:50.781013 28604 leader_election.cc:288] T 4e6854ffbc944e45b51256143505cdb8 P d1e7d0eabafa46f382e19f596e2fc00d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541), 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131:38693)
I20240502 01:52:50.781852 26490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e6854ffbc944e45b51256143505cdb8" candidate_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" is_pre_election: true
I20240502 01:52:50.782145 26490 raft_consensus.cc:2454] T 4e6854ffbc944e45b51256143505cdb8 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e7d0eabafa46f382e19f596e2fc00d in term 1.
I20240502 01:52:50.782630 26556 leader_election.cc:302] T 4e6854ffbc944e45b51256143505cdb8 P d1e7d0eabafa46f382e19f596e2fc00d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c70fbdc8a0b42ad8c09c6dbc842cc96, d1e7d0eabafa46f382e19f596e2fc00d; no voters: 
W20240502 01:52:50.782930 26556 leader_election.cc:334] T 4e6854ffbc944e45b51256143505cdb8 P d1e7d0eabafa46f382e19f596e2fc00d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111)
I20240502 01:52:50.782939 28604 raft_consensus.cc:2781] T 4e6854ffbc944e45b51256143505cdb8 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:52:50.783108 28604 raft_consensus.cc:482] T 4e6854ffbc944e45b51256143505cdb8 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Starting leader election (detected failure of leader f215e8fdc8cb45f9a5eff7cb9e6ccfce)
I20240502 01:52:50.783211 28604 raft_consensus.cc:3037] T 4e6854ffbc944e45b51256143505cdb8 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:50.784152 26077 catalog_manager.cc:5484] T fbd456fceb0f4c6db45662eb619cfd21 P 868e6e4358f4475bb45967b05eccb09f reported cstate change: term changed from 1 to 2, leader changed from f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129) to 868e6e4358f4475bb45967b05eccb09f (127.25.109.130). New cstate: current_term: 2 leader_uuid: "868e6e4358f4475bb45967b05eccb09f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:50.786154 28604 raft_consensus.cc:504] T 4e6854ffbc944e45b51256143505cdb8 P d1e7d0eabafa46f382e19f596e2fc00d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
W20240502 01:52:50.788704 26556 leader_election.cc:334] T 4e6854ffbc944e45b51256143505cdb8 P d1e7d0eabafa46f382e19f596e2fc00d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111)
I20240502 01:52:50.789126 28604 leader_election.cc:288] T 4e6854ffbc944e45b51256143505cdb8 P d1e7d0eabafa46f382e19f596e2fc00d [CANDIDATE]: Term 2 election: Requested vote from peers f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541), 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131:38693)
I20240502 01:52:50.789813 26490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e6854ffbc944e45b51256143505cdb8" candidate_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96"
I20240502 01:52:50.790150 26490 raft_consensus.cc:3037] T 4e6854ffbc944e45b51256143505cdb8 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:50.792837 26490 raft_consensus.cc:2454] T 4e6854ffbc944e45b51256143505cdb8 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e7d0eabafa46f382e19f596e2fc00d in term 2.
I20240502 01:52:50.793269 26556 leader_election.cc:302] T 4e6854ffbc944e45b51256143505cdb8 P d1e7d0eabafa46f382e19f596e2fc00d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6c70fbdc8a0b42ad8c09c6dbc842cc96, d1e7d0eabafa46f382e19f596e2fc00d; no voters: f215e8fdc8cb45f9a5eff7cb9e6ccfce
I20240502 01:52:50.793608 28604 raft_consensus.cc:2781] T 4e6854ffbc944e45b51256143505cdb8 P d1e7d0eabafa46f382e19f596e2fc00d [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:52:50.793782 28604 raft_consensus.cc:686] T 4e6854ffbc944e45b51256143505cdb8 P d1e7d0eabafa46f382e19f596e2fc00d [term 2 LEADER]: Becoming Leader. State: Replica: d1e7d0eabafa46f382e19f596e2fc00d, State: Running, Role: LEADER
I20240502 01:52:50.794137 28604 consensus_queue.cc:237] T 4e6854ffbc944e45b51256143505cdb8 P d1e7d0eabafa46f382e19f596e2fc00d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:52:50.797075 26077 catalog_manager.cc:5484] T 4e6854ffbc944e45b51256143505cdb8 P d1e7d0eabafa46f382e19f596e2fc00d reported cstate change: term changed from 1 to 2, leader changed from f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129) to d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132). New cstate: current_term: 2 leader_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:50.852272 28610 raft_consensus.cc:482] T 64c74d4af3254bc28110d5b6d3b1c51f P 868e6e4358f4475bb45967b05eccb09f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f215e8fdc8cb45f9a5eff7cb9e6ccfce)
I20240502 01:52:50.852494 28610 raft_consensus.cc:504] T 64c74d4af3254bc28110d5b6d3b1c51f P 868e6e4358f4475bb45967b05eccb09f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:50.853124 28610 leader_election.cc:288] T 64c74d4af3254bc28110d5b6d3b1c51f P 868e6e4358f4475bb45967b05eccb09f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131:38693), f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541)
I20240502 01:52:50.853782 26490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64c74d4af3254bc28110d5b6d3b1c51f" candidate_uuid: "868e6e4358f4475bb45967b05eccb09f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" is_pre_election: true
W20240502 01:52:50.854604 26292 leader_election.cc:334] T 64c74d4af3254bc28110d5b6d3b1c51f P 868e6e4358f4475bb45967b05eccb09f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111)
I20240502 01:52:50.854735 26292 leader_election.cc:302] T 64c74d4af3254bc28110d5b6d3b1c51f P 868e6e4358f4475bb45967b05eccb09f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 868e6e4358f4475bb45967b05eccb09f; no voters: 6c70fbdc8a0b42ad8c09c6dbc842cc96, f215e8fdc8cb45f9a5eff7cb9e6ccfce
I20240502 01:52:50.855031 28610 raft_consensus.cc:2726] T 64c74d4af3254bc28110d5b6d3b1c51f P 868e6e4358f4475bb45967b05eccb09f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:52:50.865190 28632 raft_consensus.cc:482] T 64c74d4af3254bc28110d5b6d3b1c51f P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f215e8fdc8cb45f9a5eff7cb9e6ccfce)
I20240502 01:52:50.865393 28632 raft_consensus.cc:504] T 64c74d4af3254bc28110d5b6d3b1c51f P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:50.866230 28632 leader_election.cc:288] T 64c74d4af3254bc28110d5b6d3b1c51f P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541), 868e6e4358f4475bb45967b05eccb09f (127.25.109.130:45571)
I20240502 01:52:50.866833 26357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64c74d4af3254bc28110d5b6d3b1c51f" candidate_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "868e6e4358f4475bb45967b05eccb09f" is_pre_election: true
I20240502 01:52:50.867089 26357 raft_consensus.cc:2454] T 64c74d4af3254bc28110d5b6d3b1c51f P 868e6e4358f4475bb45967b05eccb09f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c70fbdc8a0b42ad8c09c6dbc842cc96 in term 1.
I20240502 01:52:50.867565 26423 leader_election.cc:302] T 64c74d4af3254bc28110d5b6d3b1c51f P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c70fbdc8a0b42ad8c09c6dbc842cc96, 868e6e4358f4475bb45967b05eccb09f; no voters: 
W20240502 01:52:50.867895 26424 leader_election.cc:334] T 64c74d4af3254bc28110d5b6d3b1c51f P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111)
I20240502 01:52:50.868279 28632 raft_consensus.cc:2781] T 64c74d4af3254bc28110d5b6d3b1c51f P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:52:50.868408 28632 raft_consensus.cc:482] T 64c74d4af3254bc28110d5b6d3b1c51f P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f215e8fdc8cb45f9a5eff7cb9e6ccfce)
I20240502 01:52:50.868512 28632 raft_consensus.cc:3037] T 64c74d4af3254bc28110d5b6d3b1c51f P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:50.870784 28632 raft_consensus.cc:504] T 64c74d4af3254bc28110d5b6d3b1c51f P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:50.871445 28632 leader_election.cc:288] T 64c74d4af3254bc28110d5b6d3b1c51f P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [CANDIDATE]: Term 2 election: Requested vote from peers f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541), 868e6e4358f4475bb45967b05eccb09f (127.25.109.130:45571)
I20240502 01:52:50.872087 26357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64c74d4af3254bc28110d5b6d3b1c51f" candidate_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "868e6e4358f4475bb45967b05eccb09f"
I20240502 01:52:50.872352 26357 raft_consensus.cc:3037] T 64c74d4af3254bc28110d5b6d3b1c51f P 868e6e4358f4475bb45967b05eccb09f [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:52:50.873617 26424 leader_election.cc:334] T 64c74d4af3254bc28110d5b6d3b1c51f P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111)
I20240502 01:52:50.874964 26357 raft_consensus.cc:2454] T 64c74d4af3254bc28110d5b6d3b1c51f P 868e6e4358f4475bb45967b05eccb09f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c70fbdc8a0b42ad8c09c6dbc842cc96 in term 2.
I20240502 01:52:50.875345 26423 leader_election.cc:302] T 64c74d4af3254bc28110d5b6d3b1c51f P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6c70fbdc8a0b42ad8c09c6dbc842cc96, 868e6e4358f4475bb45967b05eccb09f; no voters: f215e8fdc8cb45f9a5eff7cb9e6ccfce
I20240502 01:52:50.875667 28632 raft_consensus.cc:2781] T 64c74d4af3254bc28110d5b6d3b1c51f P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:52:50.875833 28632 raft_consensus.cc:686] T 64c74d4af3254bc28110d5b6d3b1c51f P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 2 LEADER]: Becoming Leader. State: Replica: 6c70fbdc8a0b42ad8c09c6dbc842cc96, State: Running, Role: LEADER
I20240502 01:52:50.876164 28632 consensus_queue.cc:237] T 64c74d4af3254bc28110d5b6d3b1c51f P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:52:50.878834 26077 catalog_manager.cc:5484] T 64c74d4af3254bc28110d5b6d3b1c51f P 6c70fbdc8a0b42ad8c09c6dbc842cc96 reported cstate change: term changed from 1 to 2, leader changed from f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129) to 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131). New cstate: current_term: 2 leader_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:50.888358 28604 raft_consensus.cc:482] T ebe8589feed3400aa6bca73df420778d P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f215e8fdc8cb45f9a5eff7cb9e6ccfce)
I20240502 01:52:50.888608 28604 raft_consensus.cc:504] T ebe8589feed3400aa6bca73df420778d P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } }
I20240502 01:52:50.889313 28604 leader_election.cc:288] T ebe8589feed3400aa6bca73df420778d P d1e7d0eabafa46f382e19f596e2fc00d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541), 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131:38693)
I20240502 01:52:50.890129 26490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebe8589feed3400aa6bca73df420778d" candidate_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" is_pre_election: true
W20240502 01:52:50.890595 26556 leader_election.cc:334] T ebe8589feed3400aa6bca73df420778d P d1e7d0eabafa46f382e19f596e2fc00d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111)
I20240502 01:52:50.890898 26556 leader_election.cc:302] T ebe8589feed3400aa6bca73df420778d P d1e7d0eabafa46f382e19f596e2fc00d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1e7d0eabafa46f382e19f596e2fc00d; no voters: 6c70fbdc8a0b42ad8c09c6dbc842cc96, f215e8fdc8cb45f9a5eff7cb9e6ccfce
I20240502 01:52:50.891165 28604 raft_consensus.cc:2726] T ebe8589feed3400aa6bca73df420778d P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:52:50.894958 28632 raft_consensus.cc:482] T e425e061e59a4bffa56b498fdf462a58 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f215e8fdc8cb45f9a5eff7cb9e6ccfce)
I20240502 01:52:50.895140 28632 raft_consensus.cc:504] T e425e061e59a4bffa56b498fdf462a58 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:52:50.895730 28632 leader_election.cc:288] T e425e061e59a4bffa56b498fdf462a58 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541), d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132:37695)
I20240502 01:52:50.896318 26621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e425e061e59a4bffa56b498fdf462a58" candidate_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" is_pre_election: true
W20240502 01:52:50.897045 26424 leader_election.cc:334] T e425e061e59a4bffa56b498fdf462a58 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111)
I20240502 01:52:50.897168 26424 leader_election.cc:302] T e425e061e59a4bffa56b498fdf462a58 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c70fbdc8a0b42ad8c09c6dbc842cc96; no voters: d1e7d0eabafa46f382e19f596e2fc00d, f215e8fdc8cb45f9a5eff7cb9e6ccfce
I20240502 01:52:50.897439 28632 raft_consensus.cc:2726] T e425e061e59a4bffa56b498fdf462a58 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:52:50.909910 28632 raft_consensus.cc:482] T ebe8589feed3400aa6bca73df420778d P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f215e8fdc8cb45f9a5eff7cb9e6ccfce)
I20240502 01:52:50.910135 28632 raft_consensus.cc:504] T ebe8589feed3400aa6bca73df420778d P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } }
I20240502 01:52:50.910750 28632 leader_election.cc:288] T ebe8589feed3400aa6bca73df420778d P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132:37695), f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541)
I20240502 01:52:50.911393 26621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebe8589feed3400aa6bca73df420778d" candidate_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" is_pre_election: true
I20240502 01:52:50.911639 26621 raft_consensus.cc:2454] T ebe8589feed3400aa6bca73df420778d P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c70fbdc8a0b42ad8c09c6dbc842cc96 in term 1.
I20240502 01:52:50.912050 26423 leader_election.cc:302] T ebe8589feed3400aa6bca73df420778d P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c70fbdc8a0b42ad8c09c6dbc842cc96, d1e7d0eabafa46f382e19f596e2fc00d; no voters: 
W20240502 01:52:50.912314 26424 leader_election.cc:334] T ebe8589feed3400aa6bca73df420778d P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111)
I20240502 01:52:50.912384 28632 raft_consensus.cc:2781] T ebe8589feed3400aa6bca73df420778d P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:52:50.912497 28632 raft_consensus.cc:482] T ebe8589feed3400aa6bca73df420778d P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f215e8fdc8cb45f9a5eff7cb9e6ccfce)
I20240502 01:52:50.912604 28632 raft_consensus.cc:3037] T ebe8589feed3400aa6bca73df420778d P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:50.914455 28632 raft_consensus.cc:504] T ebe8589feed3400aa6bca73df420778d P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } }
I20240502 01:52:50.915014 28632 leader_election.cc:288] T ebe8589feed3400aa6bca73df420778d P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [CANDIDATE]: Term 2 election: Requested vote from peers d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132:37695), f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541)
I20240502 01:52:50.915467 26621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebe8589feed3400aa6bca73df420778d" candidate_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d1e7d0eabafa46f382e19f596e2fc00d"
I20240502 01:52:50.915675 26621 raft_consensus.cc:3037] T ebe8589feed3400aa6bca73df420778d P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:52:50.916306 26424 leader_election.cc:334] T ebe8589feed3400aa6bca73df420778d P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111)
I20240502 01:52:50.917768 26621 raft_consensus.cc:2454] T ebe8589feed3400aa6bca73df420778d P d1e7d0eabafa46f382e19f596e2fc00d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c70fbdc8a0b42ad8c09c6dbc842cc96 in term 2.
I20240502 01:52:50.918144 26423 leader_election.cc:302] T ebe8589feed3400aa6bca73df420778d P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6c70fbdc8a0b42ad8c09c6dbc842cc96, d1e7d0eabafa46f382e19f596e2fc00d; no voters: f215e8fdc8cb45f9a5eff7cb9e6ccfce
I20240502 01:52:50.918435 28632 raft_consensus.cc:2781] T ebe8589feed3400aa6bca73df420778d P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:52:50.918591 28632 raft_consensus.cc:686] T ebe8589feed3400aa6bca73df420778d P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 2 LEADER]: Becoming Leader. State: Replica: 6c70fbdc8a0b42ad8c09c6dbc842cc96, State: Running, Role: LEADER
I20240502 01:52:50.918848 28632 consensus_queue.cc:237] T ebe8589feed3400aa6bca73df420778d P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } }
I20240502 01:52:50.921697 26077 catalog_manager.cc:5484] T ebe8589feed3400aa6bca73df420778d P 6c70fbdc8a0b42ad8c09c6dbc842cc96 reported cstate change: term changed from 1 to 2, leader changed from f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129) to 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131). New cstate: current_term: 2 leader_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:51.043071 28604 raft_consensus.cc:482] T e425e061e59a4bffa56b498fdf462a58 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f215e8fdc8cb45f9a5eff7cb9e6ccfce)
I20240502 01:52:51.043293 28604 raft_consensus.cc:504] T e425e061e59a4bffa56b498fdf462a58 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:52:51.043985 28604 leader_election.cc:288] T e425e061e59a4bffa56b498fdf462a58 P d1e7d0eabafa46f382e19f596e2fc00d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541), 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131:38693)
I20240502 01:52:51.044947 26490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e425e061e59a4bffa56b498fdf462a58" candidate_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" is_pre_election: true
I20240502 01:52:51.045197 26490 raft_consensus.cc:2454] T e425e061e59a4bffa56b498fdf462a58 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1e7d0eabafa46f382e19f596e2fc00d in term 1.
W20240502 01:52:51.045393 26556 leader_election.cc:334] T e425e061e59a4bffa56b498fdf462a58 P d1e7d0eabafa46f382e19f596e2fc00d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111)
I20240502 01:52:51.045675 26556 leader_election.cc:302] T e425e061e59a4bffa56b498fdf462a58 P d1e7d0eabafa46f382e19f596e2fc00d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6c70fbdc8a0b42ad8c09c6dbc842cc96, d1e7d0eabafa46f382e19f596e2fc00d; no voters: f215e8fdc8cb45f9a5eff7cb9e6ccfce
I20240502 01:52:51.045939 28604 raft_consensus.cc:2781] T e425e061e59a4bffa56b498fdf462a58 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:52:51.046028 28604 raft_consensus.cc:482] T e425e061e59a4bffa56b498fdf462a58 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Starting leader election (detected failure of leader f215e8fdc8cb45f9a5eff7cb9e6ccfce)
I20240502 01:52:51.046126 28604 raft_consensus.cc:3037] T e425e061e59a4bffa56b498fdf462a58 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:51.048064 28604 raft_consensus.cc:504] T e425e061e59a4bffa56b498fdf462a58 P d1e7d0eabafa46f382e19f596e2fc00d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:52:51.048666 28604 leader_election.cc:288] T e425e061e59a4bffa56b498fdf462a58 P d1e7d0eabafa46f382e19f596e2fc00d [CANDIDATE]: Term 2 election: Requested vote from peers f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541), 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131:38693)
I20240502 01:52:51.049477 26490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e425e061e59a4bffa56b498fdf462a58" candidate_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96"
I20240502 01:52:51.049719 26490 raft_consensus.cc:3037] T e425e061e59a4bffa56b498fdf462a58 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:52:51.050024 26556 leader_election.cc:334] T e425e061e59a4bffa56b498fdf462a58 P d1e7d0eabafa46f382e19f596e2fc00d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111)
I20240502 01:52:51.052287 28604 raft_consensus.cc:1050] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d: attempting to promote NON_VOTER c292d733f9584891b46f52aafee403fc to VOTER
I20240502 01:52:51.052407 26490 raft_consensus.cc:2454] T e425e061e59a4bffa56b498fdf462a58 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e7d0eabafa46f382e19f596e2fc00d in term 2.
I20240502 01:52:51.052850 26556 leader_election.cc:302] T e425e061e59a4bffa56b498fdf462a58 P d1e7d0eabafa46f382e19f596e2fc00d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6c70fbdc8a0b42ad8c09c6dbc842cc96, d1e7d0eabafa46f382e19f596e2fc00d; no voters: f215e8fdc8cb45f9a5eff7cb9e6ccfce
I20240502 01:52:51.053169 28645 raft_consensus.cc:2781] T e425e061e59a4bffa56b498fdf462a58 P d1e7d0eabafa46f382e19f596e2fc00d [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:52:51.053332 28645 raft_consensus.cc:686] T e425e061e59a4bffa56b498fdf462a58 P d1e7d0eabafa46f382e19f596e2fc00d [term 2 LEADER]: Becoming Leader. State: Replica: d1e7d0eabafa46f382e19f596e2fc00d, State: Running, Role: LEADER
I20240502 01:52:51.053341 28604 consensus_queue.cc:237] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } }
I20240502 01:52:51.053635 28645 consensus_queue.cc:237] T e425e061e59a4bffa56b498fdf462a58 P d1e7d0eabafa46f382e19f596e2fc00d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:52:51.057165 26077 catalog_manager.cc:5484] T e425e061e59a4bffa56b498fdf462a58 P d1e7d0eabafa46f382e19f596e2fc00d reported cstate change: term changed from 1 to 2, leader changed from f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129) to d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132). New cstate: current_term: 2 leader_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } health_report { overall_health: HEALTHY } } }
W20240502 01:52:51.057868 26556 consensus_peers.cc:485] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:52:51.058521 26490 raft_consensus.cc:1261] T 7aa623407e7844989e7bf34db5d6ba35 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Refusing update from remote peer d1e7d0eabafa46f382e19f596e2fc00d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:52:51.058676 28160 raft_consensus.cc:1261] T 7aa623407e7844989e7bf34db5d6ba35 P c292d733f9584891b46f52aafee403fc [term 1 LEARNER]: Refusing update from remote peer d1e7d0eabafa46f382e19f596e2fc00d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:52:51.059759 28648 consensus_queue.cc:1035] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:52:51.060201 28604 consensus_queue.cc:1035] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:52:51.065347 28648 raft_consensus.cc:2932] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c292d733f9584891b46f52aafee403fc (127.25.109.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } }
I20240502 01:52:51.066028 26490 raft_consensus.cc:2932] T 7aa623407e7844989e7bf34db5d6ba35 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c292d733f9584891b46f52aafee403fc (127.25.109.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } }
I20240502 01:52:51.069866 28160 raft_consensus.cc:2932] T 7aa623407e7844989e7bf34db5d6ba35 P c292d733f9584891b46f52aafee403fc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c292d733f9584891b46f52aafee403fc (127.25.109.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } }
I20240502 01:52:51.070190 26077 catalog_manager.cc:5484] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d reported cstate change: config changed from index 5 to 6, c292d733f9584891b46f52aafee403fc (127.25.109.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:52:51.075747 26621 consensus_queue.cc:237] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } }
I20240502 01:52:51.078107 26490 raft_consensus.cc:1261] T 7aa623407e7844989e7bf34db5d6ba35 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Refusing update from remote peer d1e7d0eabafa46f382e19f596e2fc00d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:52:51.078125 28160 raft_consensus.cc:1261] T 7aa623407e7844989e7bf34db5d6ba35 P c292d733f9584891b46f52aafee403fc [term 1 FOLLOWER]: Refusing update from remote peer d1e7d0eabafa46f382e19f596e2fc00d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:52:51.078660 28651 consensus_queue.cc:1035] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:52:51.078969 28648 consensus_queue.cc:1035] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:52:51.081308 28651 raft_consensus.cc:2932] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } }
I20240502 01:52:51.082477 26490 raft_consensus.cc:2932] T 7aa623407e7844989e7bf34db5d6ba35 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } }
I20240502 01:52:51.086288 26067 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7aa623407e7844989e7bf34db5d6ba35 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:52:51.087054 28160 raft_consensus.cc:2932] T 7aa623407e7844989e7bf34db5d6ba35 P c292d733f9584891b46f52aafee403fc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } }
I20240502 01:52:51.088078 26080 catalog_manager.cc:5484] T 7aa623407e7844989e7bf34db5d6ba35 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 reported cstate change: config changed from index 6 to 7, VOTER f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129) evicted. New cstate: current_term: 1 leader_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } }
W20240502 01:52:51.099843 26068 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111)
W20240502 01:52:51.101400 26068 catalog_manager.cc:4573] TS f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 7aa623407e7844989e7bf34db5d6ba35: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111)
I20240502 01:52:51.201723 26490 raft_consensus.cc:1261] T fbd456fceb0f4c6db45662eb619cfd21 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 2 FOLLOWER]: Refusing update from remote peer 868e6e4358f4475bb45967b05eccb09f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:52:51.202358 28610 consensus_queue.cc:1035] T fbd456fceb0f4c6db45662eb619cfd21 P 868e6e4358f4475bb45967b05eccb09f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:52:51.208559 26292 consensus_peers.cc:485] T fbd456fceb0f4c6db45662eb619cfd21 P 868e6e4358f4475bb45967b05eccb09f -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:52:51.212522 26490 raft_consensus.cc:1261] T 4e6854ffbc944e45b51256143505cdb8 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 2 FOLLOWER]: Refusing update from remote peer d1e7d0eabafa46f382e19f596e2fc00d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:52:51.213099 28648 consensus_queue.cc:1035] T 4e6854ffbc944e45b51256143505cdb8 P d1e7d0eabafa46f382e19f596e2fc00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:52:51.218946 26556 consensus_peers.cc:485] T 4e6854ffbc944e45b51256143505cdb8 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:52:51.337275 26424 consensus_peers.cc:485] T 64c74d4af3254bc28110d5b6d3b1c51f P 6c70fbdc8a0b42ad8c09c6dbc842cc96 -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:52:51.356993 26357 raft_consensus.cc:1261] T 64c74d4af3254bc28110d5b6d3b1c51f P 868e6e4358f4475bb45967b05eccb09f [term 2 FOLLOWER]: Refusing update from remote peer 6c70fbdc8a0b42ad8c09c6dbc842cc96: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:52:51.357712 28632 consensus_queue.cc:1035] T 64c74d4af3254bc28110d5b6d3b1c51f P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:52:51.417210 26467 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:52:51.433758 27869 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:51.441493 26335 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:52:51.442190 28005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:51.462980 26490 raft_consensus.cc:1261] T e425e061e59a4bffa56b498fdf462a58 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 2 FOLLOWER]: Refusing update from remote peer d1e7d0eabafa46f382e19f596e2fc00d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:52:51.463797 28604 consensus_queue.cc:1035] T e425e061e59a4bffa56b498fdf462a58 P d1e7d0eabafa46f382e19f596e2fc00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:52:51.465534 28139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:51.475687 27736 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:52:51.489760 26556 consensus_peers.cc:485] T e425e061e59a4bffa56b498fdf462a58 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:52:51.494753 26601 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:52:51.530967 26621 raft_consensus.cc:1261] T ebe8589feed3400aa6bca73df420778d P d1e7d0eabafa46f382e19f596e2fc00d [term 2 FOLLOWER]: Refusing update from remote peer 6c70fbdc8a0b42ad8c09c6dbc842cc96: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 01:52:51.531756 28688 consensus_queue.cc:1035] T ebe8589feed3400aa6bca73df420778d P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:52:51.540213 26424 consensus_peers.cc:485] T ebe8589feed3400aa6bca73df420778d P 6c70fbdc8a0b42ad8c09c6dbc842cc96 -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:52:51.690894 26467 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:52:51.693993 27736 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:51.704185 27869 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:51.706641 26335 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:52:51.710799 28005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:51.722709 28139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:51.734433 26601 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:52:51.856871 26490 consensus_queue.cc:237] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: NON_VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: true } }
I20240502 01:52:51.859709 26621 raft_consensus.cc:1261] T 38dbb02fca694854ab66393a87c84908 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Refusing update from remote peer 6c70fbdc8a0b42ad8c09c6dbc842cc96: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:52:51.860082 26357 raft_consensus.cc:1261] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [term 1 FOLLOWER]: Refusing update from remote peer 6c70fbdc8a0b42ad8c09c6dbc842cc96: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:52:51.860481 28442 consensus_queue.cc:1035] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:52:51.860837 28688 consensus_queue.cc:1035] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:52:51.864149 28688 raft_consensus.cc:2932] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6ecf0a12e15f45a3bfa042f88c100a5e (127.25.109.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: NON_VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: true } } }
I20240502 01:52:51.869382 26621 raft_consensus.cc:2932] T 38dbb02fca694854ab66393a87c84908 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6ecf0a12e15f45a3bfa042f88c100a5e (127.25.109.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: NON_VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: true } } }
I20240502 01:52:51.869520 26078 catalog_manager.cc:5484] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 reported cstate change: config changed from index -1 to 4, NON_VOTER 6ecf0a12e15f45a3bfa042f88c100a5e (127.25.109.134) added. New cstate: current_term: 1 leader_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: NON_VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:52:51.872149 26422 consensus_peers.cc:485] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 -> Peer 6ecf0a12e15f45a3bfa042f88c100a5e (127.25.109.134:43811): Couldn't send request to peer 6ecf0a12e15f45a3bfa042f88c100a5e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 38dbb02fca694854ab66393a87c84908. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:52:51.872953 26357 raft_consensus.cc:2932] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6ecf0a12e15f45a3bfa042f88c100a5e (127.25.109.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: NON_VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: true } } }
W20240502 01:52:51.886828 26292 consensus_peers.cc:485] T 7def725255ba4448837b47ba65a6ed44 P 868e6e4358f4475bb45967b05eccb09f -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:52:51.892800 26490 consensus_queue.cc:237] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: NON_VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: true } }
I20240502 01:52:51.895534 26621 raft_consensus.cc:1261] T d4fa5ee9130340f982a6f0bcc0b28b40 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Refusing update from remote peer 6c70fbdc8a0b42ad8c09c6dbc842cc96: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:52:51.895534 26357 raft_consensus.cc:1261] T d4fa5ee9130340f982a6f0bcc0b28b40 P 868e6e4358f4475bb45967b05eccb09f [term 1 FOLLOWER]: Refusing update from remote peer 6c70fbdc8a0b42ad8c09c6dbc842cc96: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:52:51.896247 28442 consensus_queue.cc:1035] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:52:51.896636 28632 consensus_queue.cc:1035] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:52:51.899981 28442 raft_consensus.cc:2932] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER aed397efb8844970b606bca15a77cb59 (127.25.109.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: NON_VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: true } } }
I20240502 01:52:51.902158 26621 raft_consensus.cc:2932] T d4fa5ee9130340f982a6f0bcc0b28b40 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER aed397efb8844970b606bca15a77cb59 (127.25.109.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: NON_VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: true } } }
I20240502 01:52:51.902817 26357 raft_consensus.cc:2932] T d4fa5ee9130340f982a6f0bcc0b28b40 P 868e6e4358f4475bb45967b05eccb09f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER aed397efb8844970b606bca15a77cb59 (127.25.109.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: NON_VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: true } } }
W20240502 01:52:51.905300 26422 consensus_peers.cc:485] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 -> Peer aed397efb8844970b606bca15a77cb59 (127.25.109.135:39905): Couldn't send request to peer aed397efb8844970b606bca15a77cb59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d4fa5ee9130340f982a6f0bcc0b28b40. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:52:51.908740 26080 catalog_manager.cc:5484] T d4fa5ee9130340f982a6f0bcc0b28b40 P 868e6e4358f4475bb45967b05eccb09f reported cstate change: config changed from index -1 to 4, NON_VOTER aed397efb8844970b606bca15a77cb59 (127.25.109.135) added. New cstate: current_term: 1 leader_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: NON_VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: true } } }
I20240502 01:52:51.927929 26490 consensus_queue.cc:237] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: NON_VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: true } }
I20240502 01:52:51.930323 26357 raft_consensus.cc:1261] T 895cfddfe1804d34a80b8a52e00efce2 P 868e6e4358f4475bb45967b05eccb09f [term 1 FOLLOWER]: Refusing update from remote peer 6c70fbdc8a0b42ad8c09c6dbc842cc96: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:52:51.930795 26621 raft_consensus.cc:1261] T 895cfddfe1804d34a80b8a52e00efce2 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Refusing update from remote peer 6c70fbdc8a0b42ad8c09c6dbc842cc96: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:52:51.930958 28688 consensus_queue.cc:1035] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:52:51.931401 28771 consensus_queue.cc:1035] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:52:51.934489 28442 raft_consensus.cc:2932] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c292d733f9584891b46f52aafee403fc (127.25.109.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: NON_VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: true } } }
W20240502 01:52:51.936645 26422 consensus_peers.cc:485] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 -> Peer c292d733f9584891b46f52aafee403fc (127.25.109.136:38319): Couldn't send request to peer c292d733f9584891b46f52aafee403fc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 895cfddfe1804d34a80b8a52e00efce2. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:52:51.939553 26076 catalog_manager.cc:5484] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 reported cstate change: config changed from index -1 to 4, NON_VOTER c292d733f9584891b46f52aafee403fc (127.25.109.136) added. New cstate: current_term: 1 leader_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: NON_VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:51.940577 26621 raft_consensus.cc:2932] T 895cfddfe1804d34a80b8a52e00efce2 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c292d733f9584891b46f52aafee403fc (127.25.109.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: NON_VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: true } } }
I20240502 01:52:51.941138 26357 raft_consensus.cc:2932] T 895cfddfe1804d34a80b8a52e00efce2 P 868e6e4358f4475bb45967b05eccb09f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c292d733f9584891b46f52aafee403fc (127.25.109.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: NON_VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: true } } }
W20240502 01:52:52.227289 26556 consensus_peers.cc:485] T 01b7ac587d134a61b333489d937c0a21 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:52:52.266368 26292 consensus_peers.cc:485] T 062fa1dfab1046f287e2ddb1c240c4eb P 868e6e4358f4475bb45967b05eccb09f -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:52:52.317467 26068 catalog_manager.cc:4573] TS f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 7aa623407e7844989e7bf34db5d6ba35: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111)
W20240502 01:52:52.326469 26292 consensus_peers.cc:485] T c32bdf0d644e499eaa2f755fab9abb77 P 868e6e4358f4475bb45967b05eccb09f -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:52:52.332677 28813 ts_tablet_manager.cc:902] T d4fa5ee9130340f982a6f0bcc0b28b40 P aed397efb8844970b606bca15a77cb59: Initiating tablet copy from peer 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131:38693)
I20240502 01:52:52.332871 28814 ts_tablet_manager.cc:902] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc: Initiating tablet copy from peer 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131:38693)
I20240502 01:52:52.334617 28814 tablet_copy_client.cc:308] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.131:38693
I20240502 01:52:52.335207 28813 tablet_copy_client.cc:308] T d4fa5ee9130340f982a6f0bcc0b28b40 P aed397efb8844970b606bca15a77cb59: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.131:38693
W20240502 01:52:52.354364 26556 consensus_peers.cc:485] T dc44aad1f4ab4469a29d280a11268be0 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:52:52.356410 26510 tablet_copy_service.cc:140] P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Received BeginTabletCopySession request for tablet 895cfddfe1804d34a80b8a52e00efce2 from peer c292d733f9584891b46f52aafee403fc ({username='slave'} at 127.25.109.136:55333)
I20240502 01:52:52.356632 26509 tablet_copy_service.cc:140] P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Received BeginTabletCopySession request for tablet d4fa5ee9130340f982a6f0bcc0b28b40 from peer aed397efb8844970b606bca15a77cb59 ({username='slave'} at 127.25.109.135:58199)
I20240502 01:52:52.356742 26510 tablet_copy_service.cc:161] P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Beginning new tablet copy session on tablet 895cfddfe1804d34a80b8a52e00efce2 from peer c292d733f9584891b46f52aafee403fc at {username='slave'} at 127.25.109.136:55333: session id = c292d733f9584891b46f52aafee403fc-895cfddfe1804d34a80b8a52e00efce2
I20240502 01:52:52.357236 26509 tablet_copy_service.cc:161] P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Beginning new tablet copy session on tablet d4fa5ee9130340f982a6f0bcc0b28b40 from peer aed397efb8844970b606bca15a77cb59 at {username='slave'} at 127.25.109.135:58199: session id = aed397efb8844970b606bca15a77cb59-d4fa5ee9130340f982a6f0bcc0b28b40
I20240502 01:52:52.360008 26509 tablet_copy_source_session.cc:205] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:52:52.360049 26510 tablet_copy_source_session.cc:205] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:52:52.361979 28814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 895cfddfe1804d34a80b8a52e00efce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:52.366407 28813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4fa5ee9130340f982a6f0bcc0b28b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:52.367270 28814 tablet_copy_client.cc:783] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc: tablet copy: Starting download of 0 data blocks...
I20240502 01:52:52.367571 28814 tablet_copy_client.cc:647] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc: tablet copy: Starting download of 1 WAL segments...
I20240502 01:52:52.370074 28821 tablet_copy_client.cc:1002] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc: tablet copy: Injecting latency into file download: 1500
I20240502 01:52:52.380296 28813 tablet_copy_client.cc:783] T d4fa5ee9130340f982a6f0bcc0b28b40 P aed397efb8844970b606bca15a77cb59: tablet copy: Starting download of 0 data blocks...
I20240502 01:52:52.380695 28813 tablet_copy_client.cc:647] T d4fa5ee9130340f982a6f0bcc0b28b40 P aed397efb8844970b606bca15a77cb59: tablet copy: Starting download of 1 WAL segments...
I20240502 01:52:52.382825 28822 tablet_copy_client.cc:1002] T d4fa5ee9130340f982a6f0bcc0b28b40 P aed397efb8844970b606bca15a77cb59: tablet copy: Injecting latency into file download: 1500
W20240502 01:52:52.395186 26556 consensus_peers.cc:485] T 536039b70452486684f7075cf45c1851 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:52:52.433720 28836 ts_tablet_manager.cc:902] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e: Initiating tablet copy from peer 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131:38693)
I20240502 01:52:52.436333 28836 tablet_copy_client.cc:308] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.131:38693
I20240502 01:52:52.444550 26509 tablet_copy_service.cc:140] P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Received BeginTabletCopySession request for tablet 38dbb02fca694854ab66393a87c84908 from peer 6ecf0a12e15f45a3bfa042f88c100a5e ({username='slave'} at 127.25.109.134:37819)
I20240502 01:52:52.444744 26509 tablet_copy_service.cc:161] P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Beginning new tablet copy session on tablet 38dbb02fca694854ab66393a87c84908 from peer 6ecf0a12e15f45a3bfa042f88c100a5e at {username='slave'} at 127.25.109.134:37819: session id = 6ecf0a12e15f45a3bfa042f88c100a5e-38dbb02fca694854ab66393a87c84908
I20240502 01:52:52.446923 26509 tablet_copy_source_session.cc:205] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:52:52.451853 28836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38dbb02fca694854ab66393a87c84908. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:52.463382 28836 tablet_copy_client.cc:783] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e: tablet copy: Starting download of 0 data blocks...
I20240502 01:52:52.463717 28836 tablet_copy_client.cc:647] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e: tablet copy: Starting download of 1 WAL segments...
I20240502 01:52:52.465806 28842 tablet_copy_client.cc:1002] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e: tablet copy: Injecting latency into file download: 1500
W20240502 01:52:53.386418 26068 catalog_manager.cc:4573] TS f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 7aa623407e7844989e7bf34db5d6ba35: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111)
W20240502 01:52:53.513609 26556 consensus_peers.cc:485] T 4e6854ffbc944e45b51256143505cdb8 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:52:53.683323 26556 consensus_peers.cc:485] T e425e061e59a4bffa56b498fdf462a58 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:52:53.745747 26292 consensus_peers.cc:485] T fbd456fceb0f4c6db45662eb619cfd21 P 868e6e4358f4475bb45967b05eccb09f -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:52:53.812256 26424 consensus_peers.cc:485] T 64c74d4af3254bc28110d5b6d3b1c51f P 6c70fbdc8a0b42ad8c09c6dbc842cc96 -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:52:53.870522 28814 tablet_copy_client.cc:523] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:52:53.873355 28814 tablet_bootstrap.cc:492] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc: Bootstrap starting.
I20240502 01:52:53.883363 28813 tablet_copy_client.cc:523] T d4fa5ee9130340f982a6f0bcc0b28b40 P aed397efb8844970b606bca15a77cb59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:52:53.886986 28813 tablet_bootstrap.cc:492] T d4fa5ee9130340f982a6f0bcc0b28b40 P aed397efb8844970b606bca15a77cb59: Bootstrap starting.
I20240502 01:52:53.896234 28813 log.cc:824] T d4fa5ee9130340f982a6f0bcc0b28b40 P aed397efb8844970b606bca15a77cb59: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:53.907272 28814 tablet_bootstrap.cc:492] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:52:53.907680 28814 tablet_bootstrap.cc:492] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc: Bootstrap complete.
I20240502 01:52:53.907979 28814 ts_tablet_manager.cc:1367] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc: Time spent bootstrapping tablet: real 0.035s	user 0.031s	sys 0.004s
I20240502 01:52:53.908763 28814 raft_consensus.cc:348] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: NON_VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: true } }
I20240502 01:52:53.908973 28814 raft_consensus.cc:729] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c292d733f9584891b46f52aafee403fc, State: Initialized, Role: LEARNER
I20240502 01:52:53.909166 28814 consensus_queue.cc:260] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: NON_VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: true } }
I20240502 01:52:53.909844 28814 ts_tablet_manager.cc:1398] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240502 01:52:53.910754 26509 tablet_copy_service.cc:342] P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Request end of tablet copy session c292d733f9584891b46f52aafee403fc-895cfddfe1804d34a80b8a52e00efce2 received from {username='slave'} at 127.25.109.136:55333
I20240502 01:52:53.910923 26509 tablet_copy_service.cc:433] P 6c70fbdc8a0b42ad8c09c6dbc842cc96: ending tablet copy session c292d733f9584891b46f52aafee403fc-895cfddfe1804d34a80b8a52e00efce2 on tablet 895cfddfe1804d34a80b8a52e00efce2 with peer c292d733f9584891b46f52aafee403fc
I20240502 01:52:53.928244 28813 tablet_bootstrap.cc:492] T d4fa5ee9130340f982a6f0bcc0b28b40 P aed397efb8844970b606bca15a77cb59: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:52:53.928701 28813 tablet_bootstrap.cc:492] T d4fa5ee9130340f982a6f0bcc0b28b40 P aed397efb8844970b606bca15a77cb59: Bootstrap complete.
I20240502 01:52:53.929085 28813 ts_tablet_manager.cc:1367] T d4fa5ee9130340f982a6f0bcc0b28b40 P aed397efb8844970b606bca15a77cb59: Time spent bootstrapping tablet: real 0.042s	user 0.044s	sys 0.000s
I20240502 01:52:53.933435 28813 raft_consensus.cc:348] T d4fa5ee9130340f982a6f0bcc0b28b40 P aed397efb8844970b606bca15a77cb59 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: NON_VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: true } }
I20240502 01:52:53.933811 28813 raft_consensus.cc:729] T d4fa5ee9130340f982a6f0bcc0b28b40 P aed397efb8844970b606bca15a77cb59 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aed397efb8844970b606bca15a77cb59, State: Initialized, Role: LEARNER
I20240502 01:52:53.934257 28813 consensus_queue.cc:260] T d4fa5ee9130340f982a6f0bcc0b28b40 P aed397efb8844970b606bca15a77cb59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: NON_VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: true } }
I20240502 01:52:53.935598 28813 ts_tablet_manager.cc:1398] T d4fa5ee9130340f982a6f0bcc0b28b40 P aed397efb8844970b606bca15a77cb59: Time spent starting tablet: real 0.006s	user 0.007s	sys 0.000s
I20240502 01:52:53.936563 26509 tablet_copy_service.cc:342] P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Request end of tablet copy session aed397efb8844970b606bca15a77cb59-d4fa5ee9130340f982a6f0bcc0b28b40 received from {username='slave'} at 127.25.109.135:58199
I20240502 01:52:53.936681 26509 tablet_copy_service.cc:433] P 6c70fbdc8a0b42ad8c09c6dbc842cc96: ending tablet copy session aed397efb8844970b606bca15a77cb59-d4fa5ee9130340f982a6f0bcc0b28b40 on tablet d4fa5ee9130340f982a6f0bcc0b28b40 with peer aed397efb8844970b606bca15a77cb59
I20240502 01:52:53.966358 28836 tablet_copy_client.cc:523] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:52:53.970165 28836 tablet_bootstrap.cc:492] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e: Bootstrap starting.
I20240502 01:52:53.980341 28836 log.cc:824] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:54.014626 28836 tablet_bootstrap.cc:492] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:52:54.015143 28836 tablet_bootstrap.cc:492] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e: Bootstrap complete.
I20240502 01:52:54.015590 28836 ts_tablet_manager.cc:1367] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e: Time spent bootstrapping tablet: real 0.046s	user 0.032s	sys 0.012s
I20240502 01:52:54.020053 28836 raft_consensus.cc:348] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: NON_VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: true } }
I20240502 01:52:54.020502 28836 raft_consensus.cc:729] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6ecf0a12e15f45a3bfa042f88c100a5e, State: Initialized, Role: LEARNER
I20240502 01:52:54.021028 28836 consensus_queue.cc:260] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: NON_VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: true } }
I20240502 01:52:54.022749 28836 ts_tablet_manager.cc:1398] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e: Time spent starting tablet: real 0.007s	user 0.009s	sys 0.000s
I20240502 01:52:54.023707 26509 tablet_copy_service.cc:342] P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Request end of tablet copy session 6ecf0a12e15f45a3bfa042f88c100a5e-38dbb02fca694854ab66393a87c84908 received from {username='slave'} at 127.25.109.134:37819
I20240502 01:52:54.023880 26509 tablet_copy_service.cc:433] P 6c70fbdc8a0b42ad8c09c6dbc842cc96: ending tablet copy session 6ecf0a12e15f45a3bfa042f88c100a5e-38dbb02fca694854ab66393a87c84908 on tablet 38dbb02fca694854ab66393a87c84908 with peer 6ecf0a12e15f45a3bfa042f88c100a5e
W20240502 01:52:54.053866 26424 consensus_peers.cc:485] T ebe8589feed3400aa6bca73df420778d P 6c70fbdc8a0b42ad8c09c6dbc842cc96 -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:52:54.116083 27891 raft_consensus.cc:1203] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:52:54.125751 28160 raft_consensus.cc:1203] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:52:54.231406 28026 raft_consensus.cc:1203] T d4fa5ee9130340f982a6f0bcc0b28b40 P aed397efb8844970b606bca15a77cb59 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
W20240502 01:52:54.301151 26292 consensus_peers.cc:485] T 7def725255ba4448837b47ba65a6ed44 P 868e6e4358f4475bb45967b05eccb09f -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:52:54.563040 28997 raft_consensus.cc:1050] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: attempting to promote NON_VOTER 6ecf0a12e15f45a3bfa042f88c100a5e to VOTER
I20240502 01:52:54.564209 28997 consensus_queue.cc:237] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } }
I20240502 01:52:54.566864 26622 raft_consensus.cc:1261] T 38dbb02fca694854ab66393a87c84908 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Refusing update from remote peer 6c70fbdc8a0b42ad8c09c6dbc842cc96: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:52:54.566839 27891 raft_consensus.cc:1261] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 1 LEARNER]: Refusing update from remote peer 6c70fbdc8a0b42ad8c09c6dbc842cc96: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:52:54.566908 26356 raft_consensus.cc:1261] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [term 1 FOLLOWER]: Refusing update from remote peer 6c70fbdc8a0b42ad8c09c6dbc842cc96: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:52:54.567497 28997 consensus_queue.cc:1035] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:52:54.567981 28996 consensus_queue.cc:1035] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:52:54.568316 28999 consensus_queue.cc:1035] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:52:54.572803 28999 raft_consensus.cc:2932] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6ecf0a12e15f45a3bfa042f88c100a5e (127.25.109.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } }
I20240502 01:52:54.573542 26356 raft_consensus.cc:2932] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6ecf0a12e15f45a3bfa042f88c100a5e (127.25.109.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } }
I20240502 01:52:54.573698 27891 raft_consensus.cc:2932] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6ecf0a12e15f45a3bfa042f88c100a5e (127.25.109.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } }
I20240502 01:52:54.577785 26622 raft_consensus.cc:2932] T 38dbb02fca694854ab66393a87c84908 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6ecf0a12e15f45a3bfa042f88c100a5e (127.25.109.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } }
I20240502 01:52:54.579263 26076 catalog_manager.cc:5484] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f reported cstate change: config changed from index 4 to 5, 6ecf0a12e15f45a3bfa042f88c100a5e (127.25.109.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } }
I20240502 01:52:54.611065 28999 raft_consensus.cc:1050] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: attempting to promote NON_VOTER c292d733f9584891b46f52aafee403fc to VOTER
I20240502 01:52:54.611747 28999 consensus_queue.cc:237] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } }
I20240502 01:52:54.614120 26356 raft_consensus.cc:1261] T 895cfddfe1804d34a80b8a52e00efce2 P 868e6e4358f4475bb45967b05eccb09f [term 1 FOLLOWER]: Refusing update from remote peer 6c70fbdc8a0b42ad8c09c6dbc842cc96: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:52:54.614120 28160 raft_consensus.cc:1261] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc [term 1 LEARNER]: Refusing update from remote peer 6c70fbdc8a0b42ad8c09c6dbc842cc96: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:52:54.614279 26622 raft_consensus.cc:1261] T 895cfddfe1804d34a80b8a52e00efce2 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Refusing update from remote peer 6c70fbdc8a0b42ad8c09c6dbc842cc96: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:52:54.614717 28999 consensus_queue.cc:1035] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:52:54.615101 28864 consensus_queue.cc:1035] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:52:54.615366 28996 consensus_queue.cc:1035] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:52:54.618633 28864 raft_consensus.cc:2932] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c292d733f9584891b46f52aafee403fc (127.25.109.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } }
I20240502 01:52:54.619160 26356 raft_consensus.cc:2932] T 895cfddfe1804d34a80b8a52e00efce2 P 868e6e4358f4475bb45967b05eccb09f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c292d733f9584891b46f52aafee403fc (127.25.109.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } }
I20240502 01:52:54.621923 26622 raft_consensus.cc:2932] T 895cfddfe1804d34a80b8a52e00efce2 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c292d733f9584891b46f52aafee403fc (127.25.109.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } }
I20240502 01:52:54.623265 28160 raft_consensus.cc:2932] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c292d733f9584891b46f52aafee403fc (127.25.109.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } }
I20240502 01:52:54.623994 26075 catalog_manager.cc:5484] T 895cfddfe1804d34a80b8a52e00efce2 P 868e6e4358f4475bb45967b05eccb09f reported cstate change: config changed from index 4 to 5, c292d733f9584891b46f52aafee403fc (127.25.109.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } }
I20240502 01:52:54.630985 26490 consensus_queue.cc:237] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } }
I20240502 01:52:54.632722 28160 raft_consensus.cc:1261] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc [term 1 FOLLOWER]: Refusing update from remote peer 6c70fbdc8a0b42ad8c09c6dbc842cc96: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:52:54.632894 26622 raft_consensus.cc:1261] T 895cfddfe1804d34a80b8a52e00efce2 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Refusing update from remote peer 6c70fbdc8a0b42ad8c09c6dbc842cc96: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:52:54.633324 28864 consensus_queue.cc:1035] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:52:54.634424 28864 consensus_queue.cc:1035] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:52:54.635780 28996 raft_consensus.cc:2932] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 868e6e4358f4475bb45967b05eccb09f (127.25.109.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } }
I20240502 01:52:54.636324 28160 raft_consensus.cc:2932] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 868e6e4358f4475bb45967b05eccb09f (127.25.109.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } }
I20240502 01:52:54.637353 26622 raft_consensus.cc:2932] T 895cfddfe1804d34a80b8a52e00efce2 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 868e6e4358f4475bb45967b05eccb09f (127.25.109.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } }
I20240502 01:52:54.639345 26068 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 895cfddfe1804d34a80b8a52e00efce2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:52:54.641376 26080 catalog_manager.cc:5484] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 reported cstate change: config changed from index 5 to 6, VOTER 868e6e4358f4475bb45967b05eccb09f (127.25.109.130) evicted. New cstate: current_term: 1 leader_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:52:54.644811 26335 tablet_service.cc:1514] Processing DeleteTablet for tablet 895cfddfe1804d34a80b8a52e00efce2 with delete_type TABLET_DATA_TOMBSTONED (TS 868e6e4358f4475bb45967b05eccb09f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47846
I20240502 01:52:54.645689 29013 tablet_replica.cc:330] T 895cfddfe1804d34a80b8a52e00efce2 P 868e6e4358f4475bb45967b05eccb09f: stopping tablet replica
I20240502 01:52:54.646090 29013 raft_consensus.cc:2229] T 895cfddfe1804d34a80b8a52e00efce2 P 868e6e4358f4475bb45967b05eccb09f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:52:54.646448 29013 raft_consensus.cc:2258] T 895cfddfe1804d34a80b8a52e00efce2 P 868e6e4358f4475bb45967b05eccb09f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:54.648001 29013 ts_tablet_manager.cc:1875] T 895cfddfe1804d34a80b8a52e00efce2 P 868e6e4358f4475bb45967b05eccb09f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:52:54.652993 29013 ts_tablet_manager.cc:1888] T 895cfddfe1804d34a80b8a52e00efce2 P 868e6e4358f4475bb45967b05eccb09f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:52:54.653174 29013 log.cc:1196] T 895cfddfe1804d34a80b8a52e00efce2 P 868e6e4358f4475bb45967b05eccb09f: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal/wals/895cfddfe1804d34a80b8a52e00efce2
I20240502 01:52:54.654173 26067 catalog_manager.cc:4830] TS 868e6e4358f4475bb45967b05eccb09f (127.25.109.130:45571): tablet 895cfddfe1804d34a80b8a52e00efce2 (table rebalance_test_table_0 [id=45682f0e9b4f45c3b3e4cb8de40b23bd]) successfully deleted
W20240502 01:52:54.656711 26556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240502 01:52:54.657747 26556 consensus_peers.cc:485] T 01b7ac587d134a61b333489d937c0a21 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:52:54.714660 26292 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 01:52:54.715720 26292 consensus_peers.cc:485] T c32bdf0d644e499eaa2f755fab9abb77 P 868e6e4358f4475bb45967b05eccb09f -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:52:54.729703 26490 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "38dbb02fca694854ab66393a87c84908"
dest_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:40708
I20240502 01:52:54.729970 26490 raft_consensus.cc:595] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 LEADER]: Received request to transfer leadership
W20240502 01:52:54.764308 26292 consensus_peers.cc:485] T 062fa1dfab1046f287e2ddb1c240c4eb P 868e6e4358f4475bb45967b05eccb09f -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:52:54.769135 26490 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "38dbb02fca694854ab66393a87c84908"
dest_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:40744
I20240502 01:52:54.769409 26490 raft_consensus.cc:595] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:52:54.774575 28864 raft_consensus.cc:1050] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: attempting to promote NON_VOTER aed397efb8844970b606bca15a77cb59 to VOTER
I20240502 01:52:54.775492 28864 consensus_queue.cc:237] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } }
I20240502 01:52:54.786355 28026 raft_consensus.cc:1261] T d4fa5ee9130340f982a6f0bcc0b28b40 P aed397efb8844970b606bca15a77cb59 [term 1 LEARNER]: Refusing update from remote peer 6c70fbdc8a0b42ad8c09c6dbc842cc96: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:52:54.787101 26622 raft_consensus.cc:1261] T d4fa5ee9130340f982a6f0bcc0b28b40 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Refusing update from remote peer 6c70fbdc8a0b42ad8c09c6dbc842cc96: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:52:54.788108 28864 consensus_queue.cc:1035] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:52:54.788244 26356 raft_consensus.cc:1261] T d4fa5ee9130340f982a6f0bcc0b28b40 P 868e6e4358f4475bb45967b05eccb09f [term 1 FOLLOWER]: Refusing update from remote peer 6c70fbdc8a0b42ad8c09c6dbc842cc96: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:52:54.788527 28996 consensus_queue.cc:1035] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:52:54.788807 28999 consensus_queue.cc:1035] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:52:54.801299 28996 raft_consensus.cc:2932] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, aed397efb8844970b606bca15a77cb59 (127.25.109.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } }
I20240502 01:52:54.801882 26622 raft_consensus.cc:2932] T d4fa5ee9130340f982a6f0bcc0b28b40 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, aed397efb8844970b606bca15a77cb59 (127.25.109.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } }
I20240502 01:52:54.806461 26080 catalog_manager.cc:5484] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 reported cstate change: config changed from index 4 to 5, aed397efb8844970b606bca15a77cb59 (127.25.109.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:52:54.805471 28026 raft_consensus.cc:2932] T d4fa5ee9130340f982a6f0bcc0b28b40 P aed397efb8844970b606bca15a77cb59 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, aed397efb8844970b606bca15a77cb59 (127.25.109.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } }
I20240502 01:52:54.807050 26356 raft_consensus.cc:2932] T d4fa5ee9130340f982a6f0bcc0b28b40 P 868e6e4358f4475bb45967b05eccb09f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, aed397efb8844970b606bca15a77cb59 (127.25.109.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } }
I20240502 01:52:54.815992 26490 consensus_queue.cc:237] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } }
I20240502 01:52:54.818017 26356 raft_consensus.cc:1261] T d4fa5ee9130340f982a6f0bcc0b28b40 P 868e6e4358f4475bb45967b05eccb09f [term 1 FOLLOWER]: Refusing update from remote peer 6c70fbdc8a0b42ad8c09c6dbc842cc96: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:52:54.818572 28996 consensus_queue.cc:1035] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:52:54.821015 28864 raft_consensus.cc:2932] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } }
I20240502 01:52:54.821589 26356 raft_consensus.cc:2932] T d4fa5ee9130340f982a6f0bcc0b28b40 P 868e6e4358f4475bb45967b05eccb09f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } }
I20240502 01:52:54.826289 28026 raft_consensus.cc:1261] T d4fa5ee9130340f982a6f0bcc0b28b40 P aed397efb8844970b606bca15a77cb59 [term 1 FOLLOWER]: Refusing update from remote peer 6c70fbdc8a0b42ad8c09c6dbc842cc96: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:52:54.826830 26068 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d4fa5ee9130340f982a6f0bcc0b28b40 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:52:54.826918 28864 consensus_queue.cc:1035] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:52:54.828243 26075 catalog_manager.cc:5484] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 reported cstate change: config changed from index 5 to 6, VOTER d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132) evicted. New cstate: current_term: 1 leader_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:54.830417 28026 raft_consensus.cc:2932] T d4fa5ee9130340f982a6f0bcc0b28b40 P aed397efb8844970b606bca15a77cb59 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } }
I20240502 01:52:54.831961 26601 tablet_service.cc:1514] Processing DeleteTablet for tablet d4fa5ee9130340f982a6f0bcc0b28b40 with delete_type TABLET_DATA_TOMBSTONED (TS d1e7d0eabafa46f382e19f596e2fc00d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59648
I20240502 01:52:54.832907 29056 tablet_replica.cc:330] T d4fa5ee9130340f982a6f0bcc0b28b40 P d1e7d0eabafa46f382e19f596e2fc00d: stopping tablet replica
I20240502 01:52:54.833369 29056 raft_consensus.cc:2229] T d4fa5ee9130340f982a6f0bcc0b28b40 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:52:54.833882 29056 raft_consensus.cc:2258] T d4fa5ee9130340f982a6f0bcc0b28b40 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:54.835640 29056 ts_tablet_manager.cc:1875] T d4fa5ee9130340f982a6f0bcc0b28b40 P d1e7d0eabafa46f382e19f596e2fc00d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:52:54.840981 29056 ts_tablet_manager.cc:1888] T d4fa5ee9130340f982a6f0bcc0b28b40 P d1e7d0eabafa46f382e19f596e2fc00d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:52:54.841109 29056 log.cc:1196] T d4fa5ee9130340f982a6f0bcc0b28b40 P d1e7d0eabafa46f382e19f596e2fc00d: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal/wals/d4fa5ee9130340f982a6f0bcc0b28b40
I20240502 01:52:54.841833 26067 catalog_manager.cc:4830] TS d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132:37695): tablet d4fa5ee9130340f982a6f0bcc0b28b40 (table rebalance_test_table_2 [id=ce4b1d786b674a8fa90b2b56dcb5cd1e]) successfully deleted
W20240502 01:52:54.943018 26556 consensus_peers.cc:485] T dc44aad1f4ab4469a29d280a11268be0 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:52:54.947216 26556 consensus_peers.cc:485] T 536039b70452486684f7075cf45c1851 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:52:54.958753 28997 raft_consensus.cc:979] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: : Instructing follower 868e6e4358f4475bb45967b05eccb09f to start an election
I20240502 01:52:54.958969 28997 raft_consensus.cc:1067] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 LEADER]: Signalling peer 868e6e4358f4475bb45967b05eccb09f to start an election
I20240502 01:52:54.959899 26356 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "38dbb02fca694854ab66393a87c84908"
dest_uuid: "868e6e4358f4475bb45967b05eccb09f"
 from {username='slave'} at 127.25.109.131:33697
I20240502 01:52:54.960153 26356 raft_consensus.cc:482] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:52:54.960289 26356 raft_consensus.cc:3037] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:54.962314 26356 raft_consensus.cc:504] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } }
I20240502 01:52:54.963235 26356 leader_election.cc:288] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [CANDIDATE]: Term 2 election: Requested vote from peers 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131:38693), d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132:37695), 6ecf0a12e15f45a3bfa042f88c100a5e (127.25.109.134:43811)
I20240502 01:52:54.963649 26490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbb02fca694854ab66393a87c84908" candidate_uuid: "868e6e4358f4475bb45967b05eccb09f" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96"
I20240502 01:52:54.963774 26622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbb02fca694854ab66393a87c84908" candidate_uuid: "868e6e4358f4475bb45967b05eccb09f" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "d1e7d0eabafa46f382e19f596e2fc00d"
I20240502 01:52:54.963935 26490 raft_consensus.cc:3032] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:52:54.964025 26490 raft_consensus.cc:729] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6c70fbdc8a0b42ad8c09c6dbc842cc96, State: Running, Role: LEADER
I20240502 01:52:54.964028 26622 raft_consensus.cc:3037] T 38dbb02fca694854ab66393a87c84908 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:54.964399 26490 consensus_queue.cc:260] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } }
I20240502 01:52:54.964882 26490 raft_consensus.cc:3037] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:54.966681 26622 raft_consensus.cc:2454] T 38dbb02fca694854ab66393a87c84908 P d1e7d0eabafa46f382e19f596e2fc00d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 868e6e4358f4475bb45967b05eccb09f in term 2.
I20240502 01:52:54.967211 26490 raft_consensus.cc:2454] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 868e6e4358f4475bb45967b05eccb09f in term 2.
I20240502 01:52:54.967695 26292 leader_election.cc:302] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6c70fbdc8a0b42ad8c09c6dbc842cc96, 868e6e4358f4475bb45967b05eccb09f, d1e7d0eabafa46f382e19f596e2fc00d; no voters: 
I20240502 01:52:54.968068 28979 raft_consensus.cc:2781] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:52:54.968585 28979 raft_consensus.cc:686] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [term 2 LEADER]: Becoming Leader. State: Replica: 868e6e4358f4475bb45967b05eccb09f, State: Running, Role: LEADER
I20240502 01:52:54.969025 28979 consensus_queue.cc:237] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } }
I20240502 01:52:54.971673 27891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbb02fca694854ab66393a87c84908" candidate_uuid: "868e6e4358f4475bb45967b05eccb09f" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e"
I20240502 01:52:54.972002 27891 raft_consensus.cc:3037] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:54.972292 26078 catalog_manager.cc:5484] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f reported cstate change: term changed from 1 to 2, leader changed from 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131) to 868e6e4358f4475bb45967b05eccb09f (127.25.109.130). New cstate: current_term: 2 leader_uuid: "868e6e4358f4475bb45967b05eccb09f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:54.974678 27891 raft_consensus.cc:2454] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 868e6e4358f4475bb45967b05eccb09f in term 2.
I20240502 01:52:55.445103 27891 raft_consensus.cc:1261] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 2 FOLLOWER]: Refusing update from remote peer 868e6e4358f4475bb45967b05eccb09f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:52:55.445758 28979 consensus_queue.cc:1035] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 01:52:55.486559 26068 catalog_manager.cc:4573] TS f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 7aa623407e7844989e7bf34db5d6ba35: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111)
I20240502 01:52:55.537403 26490 raft_consensus.cc:1261] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 2 FOLLOWER]: Refusing update from remote peer 868e6e4358f4475bb45967b05eccb09f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:52:55.537959 28979 consensus_queue.cc:1035] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:52:55.542055 26622 raft_consensus.cc:1261] T 38dbb02fca694854ab66393a87c84908 P d1e7d0eabafa46f382e19f596e2fc00d [term 2 FOLLOWER]: Refusing update from remote peer 868e6e4358f4475bb45967b05eccb09f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:52:55.542940 28979 consensus_queue.cc:1035] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 01:52:55.800834 26424 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111) [suppressed 24 similar messages]
I20240502 01:52:55.938184 26356 consensus_queue.cc:237] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } }
I20240502 01:52:55.940047 27891 raft_consensus.cc:1261] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 2 FOLLOWER]: Refusing update from remote peer 868e6e4358f4475bb45967b05eccb09f: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:52:55.940066 26622 raft_consensus.cc:1261] T 38dbb02fca694854ab66393a87c84908 P d1e7d0eabafa46f382e19f596e2fc00d [term 2 FOLLOWER]: Refusing update from remote peer 868e6e4358f4475bb45967b05eccb09f: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:52:55.940587 28979 consensus_queue.cc:1035] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:52:55.941164 29093 consensus_queue.cc:1035] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:52:55.942868 28979 raft_consensus.cc:2932] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } }
I20240502 01:52:55.943447 27891 raft_consensus.cc:2932] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } }
I20240502 01:52:55.945711 26067 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 38dbb02fca694854ab66393a87c84908 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:52:55.947223 26622 raft_consensus.cc:2932] T 38dbb02fca694854ab66393a87c84908 P d1e7d0eabafa46f382e19f596e2fc00d [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } }
I20240502 01:52:55.947687 26075 catalog_manager.cc:5484] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e reported cstate change: config changed from index 5 to 7, VOTER 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131) evicted. New cstate: current_term: 2 leader_uuid: "868e6e4358f4475bb45967b05eccb09f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } }
I20240502 01:52:55.953874 26467 tablet_service.cc:1514] Processing DeleteTablet for tablet 38dbb02fca694854ab66393a87c84908 with delete_type TABLET_DATA_TOMBSTONED (TS 6c70fbdc8a0b42ad8c09c6dbc842cc96 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:39030
I20240502 01:52:55.954668 29111 tablet_replica.cc:330] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: stopping tablet replica
I20240502 01:52:55.955073 29111 raft_consensus.cc:2229] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:52:55.955322 29111 raft_consensus.cc:2258] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:55.956841 29111 ts_tablet_manager.cc:1875] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:52:55.961835 29111 ts_tablet_manager.cc:1888] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 01:52:55.962002 29111 log.cc:1196] T 38dbb02fca694854ab66393a87c84908 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/wals/38dbb02fca694854ab66393a87c84908
I20240502 01:52:55.962927 26068 catalog_manager.cc:4830] TS 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131:38693): tablet 38dbb02fca694854ab66393a87c84908 (table rebalance_test_table_3 [id=3dba071b1adb47bebabd07a94b201ce1]) successfully deleted
W20240502 01:52:56.031183 26556 consensus_peers.cc:485] T 4e6854ffbc944e45b51256143505cdb8 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:52:56.089471 26622 consensus_queue.cc:237] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: NON_VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: true } }
I20240502 01:52:56.092792 28160 raft_consensus.cc:1261] T 7aa623407e7844989e7bf34db5d6ba35 P c292d733f9584891b46f52aafee403fc [term 1 FOLLOWER]: Refusing update from remote peer d1e7d0eabafa46f382e19f596e2fc00d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:52:56.093022 26490 raft_consensus.cc:1261] T 7aa623407e7844989e7bf34db5d6ba35 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Refusing update from remote peer d1e7d0eabafa46f382e19f596e2fc00d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:52:56.093768 29124 consensus_queue.cc:1035] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:52:56.094261 28603 consensus_queue.cc:1035] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:52:56.097797 29125 raft_consensus.cc:2932] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 6ecf0a12e15f45a3bfa042f88c100a5e (127.25.109.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: NON_VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: true } } }
I20240502 01:52:56.098307 28160 raft_consensus.cc:2932] T 7aa623407e7844989e7bf34db5d6ba35 P c292d733f9584891b46f52aafee403fc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 6ecf0a12e15f45a3bfa042f88c100a5e (127.25.109.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: NON_VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: true } } }
W20240502 01:52:56.101207 26554 consensus_peers.cc:485] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer 6ecf0a12e15f45a3bfa042f88c100a5e (127.25.109.134:43811): Couldn't send request to peer 6ecf0a12e15f45a3bfa042f88c100a5e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7aa623407e7844989e7bf34db5d6ba35. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:52:56.100895 26490 raft_consensus.cc:2932] T 7aa623407e7844989e7bf34db5d6ba35 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 6ecf0a12e15f45a3bfa042f88c100a5e (127.25.109.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: NON_VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: true } } }
I20240502 01:52:56.103734 26080 catalog_manager.cc:5484] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d reported cstate change: config changed from index 7 to 8, NON_VOTER 6ecf0a12e15f45a3bfa042f88c100a5e (127.25.109.134) added. New cstate: current_term: 1 leader_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: NON_VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:52:56.311756 26556 consensus_peers.cc:485] T e425e061e59a4bffa56b498fdf462a58 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:52:56.312436 26292 consensus_peers.cc:485] T fbd456fceb0f4c6db45662eb619cfd21 P 868e6e4358f4475bb45967b05eccb09f -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:52:56.334214 26424 consensus_peers.cc:485] T ebe8589feed3400aa6bca73df420778d P 6c70fbdc8a0b42ad8c09c6dbc842cc96 -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:52:56.351655 26424 consensus_peers.cc:485] T 64c74d4af3254bc28110d5b6d3b1c51f P 6c70fbdc8a0b42ad8c09c6dbc842cc96 -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:52:56.627317 29148 ts_tablet_manager.cc:902] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e: Initiating tablet copy from peer d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132:37695)
I20240502 01:52:56.628574 29148 tablet_copy_client.cc:308] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.132:37695
I20240502 01:52:56.634492 26642 tablet_copy_service.cc:140] P d1e7d0eabafa46f382e19f596e2fc00d: Received BeginTabletCopySession request for tablet 7aa623407e7844989e7bf34db5d6ba35 from peer 6ecf0a12e15f45a3bfa042f88c100a5e ({username='slave'} at 127.25.109.134:39331)
I20240502 01:52:56.634660 26642 tablet_copy_service.cc:161] P d1e7d0eabafa46f382e19f596e2fc00d: Beginning new tablet copy session on tablet 7aa623407e7844989e7bf34db5d6ba35 from peer 6ecf0a12e15f45a3bfa042f88c100a5e at {username='slave'} at 127.25.109.134:39331: session id = 6ecf0a12e15f45a3bfa042f88c100a5e-7aa623407e7844989e7bf34db5d6ba35
I20240502 01:52:56.636641 26642 tablet_copy_source_session.cc:205] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:52:56.637918 29148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aa623407e7844989e7bf34db5d6ba35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:56.642736 29148 tablet_copy_client.cc:783] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e: tablet copy: Starting download of 0 data blocks...
I20240502 01:52:56.642997 29148 tablet_copy_client.cc:647] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e: tablet copy: Starting download of 1 WAL segments...
I20240502 01:52:56.644662 29152 tablet_copy_client.cc:1002] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e: tablet copy: Injecting latency into file download: 1500
W20240502 01:52:56.876490 26292 consensus_peers.cc:485] T 7def725255ba4448837b47ba65a6ed44 P 868e6e4358f4475bb45967b05eccb09f -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:52:57.016541 26292 consensus_peers.cc:485] T 062fa1dfab1046f287e2ddb1c240c4eb P 868e6e4358f4475bb45967b05eccb09f -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:52:57.131628 26556 consensus_peers.cc:485] T 01b7ac587d134a61b333489d937c0a21 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:52:57.291633 26556 consensus_peers.cc:485] T 536039b70452486684f7075cf45c1851 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:52:57.379467 26292 consensus_peers.cc:485] T c32bdf0d644e499eaa2f755fab9abb77 P 868e6e4358f4475bb45967b05eccb09f -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:52:57.412436 26556 consensus_peers.cc:485] T dc44aad1f4ab4469a29d280a11268be0 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:52:58.145053 29148 tablet_copy_client.cc:523] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:52:58.147759 29148 tablet_bootstrap.cc:492] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e: Bootstrap starting.
I20240502 01:52:58.182440 29148 tablet_bootstrap.cc:492] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:52:58.182843 29148 tablet_bootstrap.cc:492] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e: Bootstrap complete.
I20240502 01:52:58.183094 29148 ts_tablet_manager.cc:1367] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e: Time spent bootstrapping tablet: real 0.035s	user 0.034s	sys 0.001s
I20240502 01:52:58.183969 29148 raft_consensus.cc:348] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: NON_VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: true } }
I20240502 01:52:58.184197 29148 raft_consensus.cc:729] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6ecf0a12e15f45a3bfa042f88c100a5e, State: Initialized, Role: LEARNER
I20240502 01:52:58.184365 29148 consensus_queue.cc:260] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: NON_VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: true } }
I20240502 01:52:58.185060 29148 ts_tablet_manager.cc:1398] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e: Time spent starting tablet: real 0.002s	user 0.005s	sys 0.000s
I20240502 01:52:58.185992 26642 tablet_copy_service.cc:342] P d1e7d0eabafa46f382e19f596e2fc00d: Request end of tablet copy session 6ecf0a12e15f45a3bfa042f88c100a5e-7aa623407e7844989e7bf34db5d6ba35 received from {username='slave'} at 127.25.109.134:39331
I20240502 01:52:58.186118 26642 tablet_copy_service.cc:433] P d1e7d0eabafa46f382e19f596e2fc00d: ending tablet copy session 6ecf0a12e15f45a3bfa042f88c100a5e-7aa623407e7844989e7bf34db5d6ba35 on tablet 7aa623407e7844989e7bf34db5d6ba35 with peer 6ecf0a12e15f45a3bfa042f88c100a5e
I20240502 01:52:58.490254 27891 raft_consensus.cc:1203] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
W20240502 01:52:58.652925 26292 consensus_peers.cc:485] T fbd456fceb0f4c6db45662eb619cfd21 P 868e6e4358f4475bb45967b05eccb09f -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:52:58.671819 26556 consensus_peers.cc:485] T 4e6854ffbc944e45b51256143505cdb8 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:52:58.703002 26556 consensus_peers.cc:485] T e425e061e59a4bffa56b498fdf462a58 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:52:58.799263 26424 consensus_peers.cc:485] T ebe8589feed3400aa6bca73df420778d P 6c70fbdc8a0b42ad8c09c6dbc842cc96 -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:52:58.931143 26424 consensus_peers.cc:485] T 64c74d4af3254bc28110d5b6d3b1c51f P 6c70fbdc8a0b42ad8c09c6dbc842cc96 -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:52:58.969004 29219 raft_consensus.cc:1050] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d: attempting to promote NON_VOTER 6ecf0a12e15f45a3bfa042f88c100a5e to VOTER
I20240502 01:52:58.969972 29219 consensus_queue.cc:237] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } }
I20240502 01:52:58.972757 27891 raft_consensus.cc:1261] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 1 LEARNER]: Refusing update from remote peer d1e7d0eabafa46f382e19f596e2fc00d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:52:58.973057 26490 raft_consensus.cc:1261] T 7aa623407e7844989e7bf34db5d6ba35 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Refusing update from remote peer d1e7d0eabafa46f382e19f596e2fc00d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:52:58.973390 29221 consensus_queue.cc:1035] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:52:58.973717 28160 raft_consensus.cc:1261] T 7aa623407e7844989e7bf34db5d6ba35 P c292d733f9584891b46f52aafee403fc [term 1 FOLLOWER]: Refusing update from remote peer d1e7d0eabafa46f382e19f596e2fc00d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:52:58.973856 29219 consensus_queue.cc:1035] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:52:58.974319 29218 consensus_queue.cc:1035] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:52:58.977321 29221 raft_consensus.cc:2932] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 6ecf0a12e15f45a3bfa042f88c100a5e (127.25.109.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } }
I20240502 01:52:58.977933 28160 raft_consensus.cc:2932] T 7aa623407e7844989e7bf34db5d6ba35 P c292d733f9584891b46f52aafee403fc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 6ecf0a12e15f45a3bfa042f88c100a5e (127.25.109.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } }
I20240502 01:52:58.978068 27891 raft_consensus.cc:2932] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 6ecf0a12e15f45a3bfa042f88c100a5e (127.25.109.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } }
I20240502 01:52:58.982502 26078 catalog_manager.cc:5484] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e reported cstate change: config changed from index 8 to 9, 6ecf0a12e15f45a3bfa042f88c100a5e (127.25.109.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } }
I20240502 01:52:58.985358 26490 raft_consensus.cc:2932] T 7aa623407e7844989e7bf34db5d6ba35 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 6ecf0a12e15f45a3bfa042f88c100a5e (127.25.109.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } }
I20240502 01:52:59.079841 26622 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7aa623407e7844989e7bf34db5d6ba35"
dest_uuid: "d1e7d0eabafa46f382e19f596e2fc00d"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:34538
I20240502 01:52:59.080149 26622 raft_consensus.cc:595] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 LEADER]: Received request to transfer leadership
W20240502 01:52:59.290321 26292 consensus_peers.cc:485] T 7def725255ba4448837b47ba65a6ed44 P 868e6e4358f4475bb45967b05eccb09f -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:52:59.300858 26622 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7aa623407e7844989e7bf34db5d6ba35"
dest_uuid: "d1e7d0eabafa46f382e19f596e2fc00d"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:34560
I20240502 01:52:59.301084 26622 raft_consensus.cc:595] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 LEADER]: Received request to transfer leadership
I20240502 01:52:59.445958 29219 raft_consensus.cc:979] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d: : Instructing follower 6ecf0a12e15f45a3bfa042f88c100a5e to start an election
I20240502 01:52:59.446161 29221 raft_consensus.cc:1067] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 LEADER]: Signalling peer 6ecf0a12e15f45a3bfa042f88c100a5e to start an election
I20240502 01:52:59.447001 27891 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7aa623407e7844989e7bf34db5d6ba35"
dest_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e"
 from {username='slave'} at 127.25.109.132:40729
I20240502 01:52:59.447269 27891 raft_consensus.cc:482] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:52:59.447404 27891 raft_consensus.cc:3037] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:59.449498 27891 raft_consensus.cc:504] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } }
I20240502 01:52:59.451499 27891 leader_election.cc:288] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e [CANDIDATE]: Term 2 election: Requested vote from peers d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132:37695), 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131:38693), c292d733f9584891b46f52aafee403fc (127.25.109.136:38319)
I20240502 01:52:59.451629 26622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aa623407e7844989e7bf34db5d6ba35" candidate_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "d1e7d0eabafa46f382e19f596e2fc00d"
I20240502 01:52:59.451938 26622 raft_consensus.cc:3032] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:52:59.452039 26622 raft_consensus.cc:729] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d1e7d0eabafa46f382e19f596e2fc00d, State: Running, Role: LEADER
I20240502 01:52:59.451969 26490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aa623407e7844989e7bf34db5d6ba35" candidate_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96"
I20240502 01:52:59.452193 26490 raft_consensus.cc:3037] T 7aa623407e7844989e7bf34db5d6ba35 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:59.452420 26622 consensus_queue.cc:260] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } }
I20240502 01:52:59.452886 26622 raft_consensus.cc:3037] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:59.455047 26490 raft_consensus.cc:2454] T 7aa623407e7844989e7bf34db5d6ba35 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ecf0a12e15f45a3bfa042f88c100a5e in term 2.
I20240502 01:52:59.455619 26622 raft_consensus.cc:2454] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ecf0a12e15f45a3bfa042f88c100a5e in term 2.
I20240502 01:52:59.456344 27824 leader_election.cc:302] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6c70fbdc8a0b42ad8c09c6dbc842cc96, 6ecf0a12e15f45a3bfa042f88c100a5e, d1e7d0eabafa46f382e19f596e2fc00d; no voters: 
I20240502 01:52:59.456779 29194 raft_consensus.cc:2781] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:52:59.457388 29194 raft_consensus.cc:686] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 2 LEADER]: Becoming Leader. State: Replica: 6ecf0a12e15f45a3bfa042f88c100a5e, State: Running, Role: LEADER
I20240502 01:52:59.457948 29194 consensus_queue.cc:237] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } }
I20240502 01:52:59.462394 26080 catalog_manager.cc:5484] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e reported cstate change: term changed from 1 to 2, leader changed from d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132) to 6ecf0a12e15f45a3bfa042f88c100a5e (127.25.109.134). New cstate: current_term: 2 leader_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:52:59.463069 28160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aa623407e7844989e7bf34db5d6ba35" candidate_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "c292d733f9584891b46f52aafee403fc"
I20240502 01:52:59.463420 28160 raft_consensus.cc:3037] T 7aa623407e7844989e7bf34db5d6ba35 P c292d733f9584891b46f52aafee403fc [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:52:59.465636 28160 raft_consensus.cc:2454] T 7aa623407e7844989e7bf34db5d6ba35 P c292d733f9584891b46f52aafee403fc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ecf0a12e15f45a3bfa042f88c100a5e in term 2.
W20240502 01:52:59.595968 26292 consensus_peers.cc:485] T 062fa1dfab1046f287e2ddb1c240c4eb P 868e6e4358f4475bb45967b05eccb09f -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:52:59.605576 26068 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20240502 01:52:59.606952 26068 catalog_manager.cc:4573] TS f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 7aa623407e7844989e7bf34db5d6ba35: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111)
W20240502 01:52:59.676769 26556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 01:52:59.677795 26556 consensus_peers.cc:485] T 01b7ac587d134a61b333489d937c0a21 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:52:59.815989 26292 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 01:52:59.864291 26556 consensus_peers.cc:485] T 536039b70452486684f7075cf45c1851 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:52:59.890303 26490 raft_consensus.cc:1261] T 7aa623407e7844989e7bf34db5d6ba35 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 2 FOLLOWER]: Refusing update from remote peer 6ecf0a12e15f45a3bfa042f88c100a5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:52:59.890976 29194 consensus_queue.cc:1035] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:52:59.916173 28160 raft_consensus.cc:1261] T 7aa623407e7844989e7bf34db5d6ba35 P c292d733f9584891b46f52aafee403fc [term 2 FOLLOWER]: Refusing update from remote peer 6ecf0a12e15f45a3bfa042f88c100a5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:52:59.916796 29194 consensus_queue.cc:1035] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:52:59.921200 26622 raft_consensus.cc:1261] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [term 2 FOLLOWER]: Refusing update from remote peer 6ecf0a12e15f45a3bfa042f88c100a5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:52:59.921927 29268 consensus_queue.cc:1035] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20240502 01:52:59.935211 27891 consensus_queue.cc:237] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } }
I20240502 01:52:59.937389 26490 raft_consensus.cc:1261] T 7aa623407e7844989e7bf34db5d6ba35 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 2 FOLLOWER]: Refusing update from remote peer 6ecf0a12e15f45a3bfa042f88c100a5e: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 01:52:59.937763 28160 raft_consensus.cc:1261] T 7aa623407e7844989e7bf34db5d6ba35 P c292d733f9584891b46f52aafee403fc [term 2 FOLLOWER]: Refusing update from remote peer 6ecf0a12e15f45a3bfa042f88c100a5e: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 01:52:59.938037 29194 consensus_queue.cc:1035] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:52:59.938344 29268 consensus_queue.cc:1035] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:52:59.940732 29252 raft_consensus.cc:2932] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } }
I20240502 01:52:59.941386 28160 raft_consensus.cc:2932] T 7aa623407e7844989e7bf34db5d6ba35 P c292d733f9584891b46f52aafee403fc [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } }
I20240502 01:52:59.941563 26490 raft_consensus.cc:2932] T 7aa623407e7844989e7bf34db5d6ba35 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } }
I20240502 01:52:59.945847 26066 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7aa623407e7844989e7bf34db5d6ba35 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:52:59.946547 26080 catalog_manager.cc:5484] T 7aa623407e7844989e7bf34db5d6ba35 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 reported cstate change: config changed from index 9 to 11, VOTER d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132) evicted. New cstate: current_term: 2 leader_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } }
I20240502 01:52:59.950526 26601 tablet_service.cc:1514] Processing DeleteTablet for tablet 7aa623407e7844989e7bf34db5d6ba35 with delete_type TABLET_DATA_TOMBSTONED (TS d1e7d0eabafa46f382e19f596e2fc00d not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:59648
I20240502 01:52:59.951180 29278 tablet_replica.cc:330] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d: stopping tablet replica
I20240502 01:52:59.951443 29278 raft_consensus.cc:2229] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:52:59.952512 29278 raft_consensus.cc:2258] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:52:59.953387 29278 ts_tablet_manager.cc:1875] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:52:59.958043 29278 ts_tablet_manager.cc:1888] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240502 01:52:59.958173 29278 log.cc:1196] T 7aa623407e7844989e7bf34db5d6ba35 P d1e7d0eabafa46f382e19f596e2fc00d: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal/wals/7aa623407e7844989e7bf34db5d6ba35
I20240502 01:52:59.958940 26067 catalog_manager.cc:4830] TS d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132:37695): tablet 7aa623407e7844989e7bf34db5d6ba35 (table rebalance_test_table_4 [id=690e246e360e425598dedef6d62ce625]) successfully deleted
W20240502 01:53:00.061703 26292 consensus_peers.cc:485] T c32bdf0d644e499eaa2f755fab9abb77 P 868e6e4358f4475bb45967b05eccb09f -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:53:00.192389 26556 consensus_peers.cc:485] T dc44aad1f4ab4469a29d280a11268be0 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:53:00.259534 26467 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:53:00.263093 27869 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:00.285720 26335 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:53:00.295768 28005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:00.330756 28139 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:00.346201 27736 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:00.355216 26601 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:53:00.416508 26490 consensus_queue.cc:237] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: true } }
I20240502 01:53:00.419253 26356 raft_consensus.cc:1261] T d4fa5ee9130340f982a6f0bcc0b28b40 P 868e6e4358f4475bb45967b05eccb09f [term 1 FOLLOWER]: Refusing update from remote peer 6c70fbdc8a0b42ad8c09c6dbc842cc96: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:53:00.419239 28026 raft_consensus.cc:1261] T d4fa5ee9130340f982a6f0bcc0b28b40 P aed397efb8844970b606bca15a77cb59 [term 1 FOLLOWER]: Refusing update from remote peer 6c70fbdc8a0b42ad8c09c6dbc842cc96: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:53:00.419920 29087 consensus_queue.cc:1035] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:53:00.420403 29330 consensus_queue.cc:1035] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:53:00.422816 29087 raft_consensus.cc:2932] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ca931e9360d44079b6059b8d9e7a6d20 (127.25.109.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: true } } }
I20240502 01:53:00.423552 26356 raft_consensus.cc:2932] T d4fa5ee9130340f982a6f0bcc0b28b40 P 868e6e4358f4475bb45967b05eccb09f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ca931e9360d44079b6059b8d9e7a6d20 (127.25.109.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: true } } }
I20240502 01:53:00.425801 28026 raft_consensus.cc:2932] T d4fa5ee9130340f982a6f0bcc0b28b40 P aed397efb8844970b606bca15a77cb59 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ca931e9360d44079b6059b8d9e7a6d20 (127.25.109.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: true } } }
W20240502 01:53:00.426437 26424 consensus_peers.cc:485] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 -> Peer ca931e9360d44079b6059b8d9e7a6d20 (127.25.109.133:33081): Couldn't send request to peer ca931e9360d44079b6059b8d9e7a6d20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d4fa5ee9130340f982a6f0bcc0b28b40. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:00.428874 26080 catalog_manager.cc:5484] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 reported cstate change: config changed from index 6 to 7, NON_VOTER ca931e9360d44079b6059b8d9e7a6d20 (127.25.109.133) added. New cstate: current_term: 1 leader_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:00.444201 26356 consensus_queue.cc:237] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: NON_VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: true } }
I20240502 01:53:00.447153 27891 raft_consensus.cc:1261] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 2 FOLLOWER]: Refusing update from remote peer 868e6e4358f4475bb45967b05eccb09f: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 01:53:00.447340 26622 raft_consensus.cc:1261] T 38dbb02fca694854ab66393a87c84908 P d1e7d0eabafa46f382e19f596e2fc00d [term 2 FOLLOWER]: Refusing update from remote peer 868e6e4358f4475bb45967b05eccb09f: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 01:53:00.447701 29212 consensus_queue.cc:1035] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:53:00.448184 29342 consensus_queue.cc:1035] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:53:00.451421 29212 raft_consensus.cc:2932] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER aed397efb8844970b606bca15a77cb59 (127.25.109.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: NON_VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: true } } }
W20240502 01:53:00.455229 26290 consensus_peers.cc:485] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f -> Peer aed397efb8844970b606bca15a77cb59 (127.25.109.135:39905): Couldn't send request to peer aed397efb8844970b606bca15a77cb59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 38dbb02fca694854ab66393a87c84908. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:00.455682 26622 raft_consensus.cc:2932] T 38dbb02fca694854ab66393a87c84908 P d1e7d0eabafa46f382e19f596e2fc00d [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER aed397efb8844970b606bca15a77cb59 (127.25.109.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: NON_VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: true } } }
I20240502 01:53:00.457120 26080 catalog_manager.cc:5484] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f reported cstate change: config changed from index 7 to 8, NON_VOTER aed397efb8844970b606bca15a77cb59 (127.25.109.135) added. New cstate: current_term: 2 leader_uuid: "868e6e4358f4475bb45967b05eccb09f" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: NON_VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:00.466275 27891 raft_consensus.cc:2932] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER aed397efb8844970b606bca15a77cb59 (127.25.109.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: NON_VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: true } } }
I20240502 01:53:00.475842 26490 consensus_queue.cc:237] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: NON_VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: true } }
I20240502 01:53:00.481541 26622 raft_consensus.cc:1261] T 895cfddfe1804d34a80b8a52e00efce2 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Refusing update from remote peer 6c70fbdc8a0b42ad8c09c6dbc842cc96: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:53:00.481590 28160 raft_consensus.cc:1261] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc [term 1 FOLLOWER]: Refusing update from remote peer 6c70fbdc8a0b42ad8c09c6dbc842cc96: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:53:00.482208 29087 consensus_queue.cc:1035] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:53:00.482601 29331 consensus_queue.cc:1035] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 01:53:00.482918 26422 consensus_peers.cc:485] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 -> Peer 6ecf0a12e15f45a3bfa042f88c100a5e (127.25.109.134:43811): Couldn't send request to peer 6ecf0a12e15f45a3bfa042f88c100a5e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 895cfddfe1804d34a80b8a52e00efce2. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:00.485417 29330 raft_consensus.cc:2932] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6ecf0a12e15f45a3bfa042f88c100a5e (127.25.109.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: NON_VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: true } } }
I20240502 01:53:00.486136 26622 raft_consensus.cc:2932] T 895cfddfe1804d34a80b8a52e00efce2 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6ecf0a12e15f45a3bfa042f88c100a5e (127.25.109.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: NON_VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: true } } }
I20240502 01:53:00.486225 28160 raft_consensus.cc:2932] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6ecf0a12e15f45a3bfa042f88c100a5e (127.25.109.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: NON_VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: true } } }
I20240502 01:53:00.491451 26075 catalog_manager.cc:5484] T 895cfddfe1804d34a80b8a52e00efce2 P d1e7d0eabafa46f382e19f596e2fc00d reported cstate change: config changed from index 6 to 7, NON_VOTER 6ecf0a12e15f45a3bfa042f88c100a5e (127.25.109.134) added. New cstate: current_term: 1 leader_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: NON_VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: true } } }
I20240502 01:53:00.889842 29388 ts_tablet_manager.cc:902] T d4fa5ee9130340f982a6f0bcc0b28b40 P ca931e9360d44079b6059b8d9e7a6d20: Initiating tablet copy from peer 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131:38693)
I20240502 01:53:00.891516 29388 tablet_copy_client.cc:308] T d4fa5ee9130340f982a6f0bcc0b28b40 P ca931e9360d44079b6059b8d9e7a6d20: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.131:38693
I20240502 01:53:00.897593 26509 tablet_copy_service.cc:140] P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Received BeginTabletCopySession request for tablet d4fa5ee9130340f982a6f0bcc0b28b40 from peer ca931e9360d44079b6059b8d9e7a6d20 ({username='slave'} at 127.25.109.133:46555)
I20240502 01:53:00.897744 26509 tablet_copy_service.cc:161] P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Beginning new tablet copy session on tablet d4fa5ee9130340f982a6f0bcc0b28b40 from peer ca931e9360d44079b6059b8d9e7a6d20 at {username='slave'} at 127.25.109.133:46555: session id = ca931e9360d44079b6059b8d9e7a6d20-d4fa5ee9130340f982a6f0bcc0b28b40
I20240502 01:53:00.899739 26509 tablet_copy_source_session.cc:205] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:00.904450 29388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4fa5ee9130340f982a6f0bcc0b28b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:00.915122 29388 tablet_copy_client.cc:783] T d4fa5ee9130340f982a6f0bcc0b28b40 P ca931e9360d44079b6059b8d9e7a6d20: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:00.915424 29388 tablet_copy_client.cc:647] T d4fa5ee9130340f982a6f0bcc0b28b40 P ca931e9360d44079b6059b8d9e7a6d20: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:00.917356 29392 tablet_copy_client.cc:1002] T d4fa5ee9130340f982a6f0bcc0b28b40 P ca931e9360d44079b6059b8d9e7a6d20: tablet copy: Injecting latency into file download: 1500
I20240502 01:53:00.971372 29408 ts_tablet_manager.cc:902] T 895cfddfe1804d34a80b8a52e00efce2 P 6ecf0a12e15f45a3bfa042f88c100a5e: Initiating tablet copy from peer 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131:38693)
I20240502 01:53:00.972868 29408 tablet_copy_client.cc:308] T 895cfddfe1804d34a80b8a52e00efce2 P 6ecf0a12e15f45a3bfa042f88c100a5e: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.131:38693
I20240502 01:53:00.973608 26509 tablet_copy_service.cc:140] P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Received BeginTabletCopySession request for tablet 895cfddfe1804d34a80b8a52e00efce2 from peer 6ecf0a12e15f45a3bfa042f88c100a5e ({username='slave'} at 127.25.109.134:37819)
I20240502 01:53:00.973783 26509 tablet_copy_service.cc:161] P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Beginning new tablet copy session on tablet 895cfddfe1804d34a80b8a52e00efce2 from peer 6ecf0a12e15f45a3bfa042f88c100a5e at {username='slave'} at 127.25.109.134:37819: session id = 6ecf0a12e15f45a3bfa042f88c100a5e-895cfddfe1804d34a80b8a52e00efce2
I20240502 01:53:00.976109 26509 tablet_copy_source_session.cc:205] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:00.977506 29408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 895cfddfe1804d34a80b8a52e00efce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:00.980861 29414 ts_tablet_manager.cc:902] T 38dbb02fca694854ab66393a87c84908 P aed397efb8844970b606bca15a77cb59: Initiating tablet copy from peer 868e6e4358f4475bb45967b05eccb09f (127.25.109.130:45571)
I20240502 01:53:00.982277 29414 tablet_copy_client.cc:308] T 38dbb02fca694854ab66393a87c84908 P aed397efb8844970b606bca15a77cb59: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.130:45571
I20240502 01:53:00.983436 29408 tablet_copy_client.cc:783] T 895cfddfe1804d34a80b8a52e00efce2 P 6ecf0a12e15f45a3bfa042f88c100a5e: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:00.983757 29408 tablet_copy_client.cc:647] T 895cfddfe1804d34a80b8a52e00efce2 P 6ecf0a12e15f45a3bfa042f88c100a5e: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:00.985709 29413 tablet_copy_client.cc:1002] T 895cfddfe1804d34a80b8a52e00efce2 P 6ecf0a12e15f45a3bfa042f88c100a5e: tablet copy: Injecting latency into file download: 1500
I20240502 01:53:00.993969 26377 tablet_copy_service.cc:140] P 868e6e4358f4475bb45967b05eccb09f: Received BeginTabletCopySession request for tablet 38dbb02fca694854ab66393a87c84908 from peer aed397efb8844970b606bca15a77cb59 ({username='slave'} at 127.25.109.135:48865)
I20240502 01:53:00.994199 26377 tablet_copy_service.cc:161] P 868e6e4358f4475bb45967b05eccb09f: Beginning new tablet copy session on tablet 38dbb02fca694854ab66393a87c84908 from peer aed397efb8844970b606bca15a77cb59 at {username='slave'} at 127.25.109.135:48865: session id = aed397efb8844970b606bca15a77cb59-38dbb02fca694854ab66393a87c84908
I20240502 01:53:00.996764 26377 tablet_copy_source_session.cc:205] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:00.998539 29414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38dbb02fca694854ab66393a87c84908. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:01.002976 29414 tablet_copy_client.cc:783] T 38dbb02fca694854ab66393a87c84908 P aed397efb8844970b606bca15a77cb59: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:01.003211 29414 tablet_copy_client.cc:647] T 38dbb02fca694854ab66393a87c84908 P aed397efb8844970b606bca15a77cb59: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:01.005132 29418 tablet_copy_client.cc:1002] T 38dbb02fca694854ab66393a87c84908 P aed397efb8844970b606bca15a77cb59: tablet copy: Injecting latency into file download: 1500
W20240502 01:53:01.029819 26556 consensus_peers.cc:485] T 4e6854ffbc944e45b51256143505cdb8 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:53:01.143616 26556 consensus_peers.cc:485] T e425e061e59a4bffa56b498fdf462a58 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:53:01.201155 26424 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 01:53:01.202049 26424 consensus_peers.cc:485] T 64c74d4af3254bc28110d5b6d3b1c51f P 6c70fbdc8a0b42ad8c09c6dbc842cc96 -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:53:01.241416 26424 consensus_peers.cc:485] T ebe8589feed3400aa6bca73df420778d P 6c70fbdc8a0b42ad8c09c6dbc842cc96 -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:53:01.370520 26292 consensus_peers.cc:485] T fbd456fceb0f4c6db45662eb619cfd21 P 868e6e4358f4475bb45967b05eccb09f -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:53:01.840199 26292 consensus_peers.cc:485] T 7def725255ba4448837b47ba65a6ed44 P 868e6e4358f4475bb45967b05eccb09f -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:53:02.048406 26556 consensus_peers.cc:485] T 01b7ac587d134a61b333489d937c0a21 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:53:02.260468 26292 consensus_peers.cc:485] T 062fa1dfab1046f287e2ddb1c240c4eb P 868e6e4358f4475bb45967b05eccb09f -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:53:02.418035 29388 tablet_copy_client.cc:523] T d4fa5ee9130340f982a6f0bcc0b28b40 P ca931e9360d44079b6059b8d9e7a6d20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:02.421866 29388 tablet_bootstrap.cc:492] T d4fa5ee9130340f982a6f0bcc0b28b40 P ca931e9360d44079b6059b8d9e7a6d20: Bootstrap starting.
I20240502 01:53:02.431622 29388 log.cc:824] T d4fa5ee9130340f982a6f0bcc0b28b40 P ca931e9360d44079b6059b8d9e7a6d20: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:02.464402 29388 tablet_bootstrap.cc:492] T d4fa5ee9130340f982a6f0bcc0b28b40 P ca931e9360d44079b6059b8d9e7a6d20: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:02.464897 29388 tablet_bootstrap.cc:492] T d4fa5ee9130340f982a6f0bcc0b28b40 P ca931e9360d44079b6059b8d9e7a6d20: Bootstrap complete.
I20240502 01:53:02.465317 29388 ts_tablet_manager.cc:1367] T d4fa5ee9130340f982a6f0bcc0b28b40 P ca931e9360d44079b6059b8d9e7a6d20: Time spent bootstrapping tablet: real 0.044s	user 0.043s	sys 0.000s
I20240502 01:53:02.470578 29388 raft_consensus.cc:348] T d4fa5ee9130340f982a6f0bcc0b28b40 P ca931e9360d44079b6059b8d9e7a6d20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: true } }
I20240502 01:53:02.471032 29388 raft_consensus.cc:729] T d4fa5ee9130340f982a6f0bcc0b28b40 P ca931e9360d44079b6059b8d9e7a6d20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ca931e9360d44079b6059b8d9e7a6d20, State: Initialized, Role: LEARNER
I20240502 01:53:02.471529 29388 consensus_queue.cc:260] T d4fa5ee9130340f982a6f0bcc0b28b40 P ca931e9360d44079b6059b8d9e7a6d20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: true } }
I20240502 01:53:02.472985 29388 ts_tablet_manager.cc:1398] T d4fa5ee9130340f982a6f0bcc0b28b40 P ca931e9360d44079b6059b8d9e7a6d20: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.004s
I20240502 01:53:02.474041 26509 tablet_copy_service.cc:342] P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Request end of tablet copy session ca931e9360d44079b6059b8d9e7a6d20-d4fa5ee9130340f982a6f0bcc0b28b40 received from {username='slave'} at 127.25.109.133:46555
I20240502 01:53:02.474184 26509 tablet_copy_service.cc:433] P 6c70fbdc8a0b42ad8c09c6dbc842cc96: ending tablet copy session ca931e9360d44079b6059b8d9e7a6d20-d4fa5ee9130340f982a6f0bcc0b28b40 on tablet d4fa5ee9130340f982a6f0bcc0b28b40 with peer ca931e9360d44079b6059b8d9e7a6d20
I20240502 01:53:02.486198 29408 tablet_copy_client.cc:523] T 895cfddfe1804d34a80b8a52e00efce2 P 6ecf0a12e15f45a3bfa042f88c100a5e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:02.489094 29408 tablet_bootstrap.cc:492] T 895cfddfe1804d34a80b8a52e00efce2 P 6ecf0a12e15f45a3bfa042f88c100a5e: Bootstrap starting.
I20240502 01:53:02.505573 29414 tablet_copy_client.cc:523] T 38dbb02fca694854ab66393a87c84908 P aed397efb8844970b606bca15a77cb59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:02.509666 29414 tablet_bootstrap.cc:492] T 38dbb02fca694854ab66393a87c84908 P aed397efb8844970b606bca15a77cb59: Bootstrap starting.
I20240502 01:53:02.532336 29408 tablet_bootstrap.cc:492] T 895cfddfe1804d34a80b8a52e00efce2 P 6ecf0a12e15f45a3bfa042f88c100a5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:02.532863 29408 tablet_bootstrap.cc:492] T 895cfddfe1804d34a80b8a52e00efce2 P 6ecf0a12e15f45a3bfa042f88c100a5e: Bootstrap complete.
I20240502 01:53:02.533180 29408 ts_tablet_manager.cc:1367] T 895cfddfe1804d34a80b8a52e00efce2 P 6ecf0a12e15f45a3bfa042f88c100a5e: Time spent bootstrapping tablet: real 0.044s	user 0.042s	sys 0.002s
I20240502 01:53:02.534070 29408 raft_consensus.cc:348] T 895cfddfe1804d34a80b8a52e00efce2 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: NON_VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: true } }
I20240502 01:53:02.534379 29408 raft_consensus.cc:729] T 895cfddfe1804d34a80b8a52e00efce2 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6ecf0a12e15f45a3bfa042f88c100a5e, State: Initialized, Role: LEARNER
I20240502 01:53:02.534633 29408 consensus_queue.cc:260] T 895cfddfe1804d34a80b8a52e00efce2 P 6ecf0a12e15f45a3bfa042f88c100a5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: NON_VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: true } }
I20240502 01:53:02.535334 29408 ts_tablet_manager.cc:1398] T 895cfddfe1804d34a80b8a52e00efce2 P 6ecf0a12e15f45a3bfa042f88c100a5e: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 01:53:02.536363 26509 tablet_copy_service.cc:342] P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Request end of tablet copy session 6ecf0a12e15f45a3bfa042f88c100a5e-895cfddfe1804d34a80b8a52e00efce2 received from {username='slave'} at 127.25.109.134:37819
I20240502 01:53:02.536487 26509 tablet_copy_service.cc:433] P 6c70fbdc8a0b42ad8c09c6dbc842cc96: ending tablet copy session 6ecf0a12e15f45a3bfa042f88c100a5e-895cfddfe1804d34a80b8a52e00efce2 on tablet 895cfddfe1804d34a80b8a52e00efce2 with peer 6ecf0a12e15f45a3bfa042f88c100a5e
W20240502 01:53:02.543908 26556 consensus_peers.cc:485] T 536039b70452486684f7075cf45c1851 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:53:02.554333 29414 tablet_bootstrap.cc:492] T 38dbb02fca694854ab66393a87c84908 P aed397efb8844970b606bca15a77cb59: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:02.554745 29414 tablet_bootstrap.cc:492] T 38dbb02fca694854ab66393a87c84908 P aed397efb8844970b606bca15a77cb59: Bootstrap complete.
I20240502 01:53:02.554988 29414 ts_tablet_manager.cc:1367] T 38dbb02fca694854ab66393a87c84908 P aed397efb8844970b606bca15a77cb59: Time spent bootstrapping tablet: real 0.045s	user 0.039s	sys 0.004s
I20240502 01:53:02.556154 29414 raft_consensus.cc:348] T 38dbb02fca694854ab66393a87c84908 P aed397efb8844970b606bca15a77cb59 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: NON_VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: true } }
I20240502 01:53:02.556475 29414 raft_consensus.cc:729] T 38dbb02fca694854ab66393a87c84908 P aed397efb8844970b606bca15a77cb59 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: aed397efb8844970b606bca15a77cb59, State: Initialized, Role: LEARNER
I20240502 01:53:02.556690 29414 consensus_queue.cc:260] T 38dbb02fca694854ab66393a87c84908 P aed397efb8844970b606bca15a77cb59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: NON_VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: true } }
I20240502 01:53:02.557531 29414 ts_tablet_manager.cc:1398] T 38dbb02fca694854ab66393a87c84908 P aed397efb8844970b606bca15a77cb59: Time spent starting tablet: real 0.002s	user 0.005s	sys 0.000s
I20240502 01:53:02.558490 26377 tablet_copy_service.cc:342] P 868e6e4358f4475bb45967b05eccb09f: Request end of tablet copy session aed397efb8844970b606bca15a77cb59-38dbb02fca694854ab66393a87c84908 received from {username='slave'} at 127.25.109.135:48865
I20240502 01:53:02.558669 26377 tablet_copy_service.cc:433] P 868e6e4358f4475bb45967b05eccb09f: ending tablet copy session aed397efb8844970b606bca15a77cb59-38dbb02fca694854ab66393a87c84908 on tablet 38dbb02fca694854ab66393a87c84908 with peer aed397efb8844970b606bca15a77cb59
W20240502 01:53:02.580650 26292 consensus_peers.cc:485] T c32bdf0d644e499eaa2f755fab9abb77 P 868e6e4358f4475bb45967b05eccb09f -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:53:02.749524 27891 raft_consensus.cc:1203] T 895cfddfe1804d34a80b8a52e00efce2 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 01:53:02.812233 28026 raft_consensus.cc:1203] T 38dbb02fca694854ab66393a87c84908 P aed397efb8844970b606bca15a77cb59 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8]   Dedup: 2.8->[]
I20240502 01:53:02.867136 27757 raft_consensus.cc:1203] T d4fa5ee9130340f982a6f0bcc0b28b40 P ca931e9360d44079b6059b8d9e7a6d20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
W20240502 01:53:02.876775 26556 consensus_peers.cc:485] T dc44aad1f4ab4469a29d280a11268be0 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:53:03.160192 29571 raft_consensus.cc:1050] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: attempting to promote NON_VOTER 6ecf0a12e15f45a3bfa042f88c100a5e to VOTER
I20240502 01:53:03.160984 29571 consensus_queue.cc:237] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } }
I20240502 01:53:03.164134 27891 raft_consensus.cc:1261] T 895cfddfe1804d34a80b8a52e00efce2 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 1 LEARNER]: Refusing update from remote peer 6c70fbdc8a0b42ad8c09c6dbc842cc96: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:53:03.164218 28160 raft_consensus.cc:1261] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc [term 1 FOLLOWER]: Refusing update from remote peer 6c70fbdc8a0b42ad8c09c6dbc842cc96: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:53:03.164608 26622 raft_consensus.cc:1261] T 895cfddfe1804d34a80b8a52e00efce2 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Refusing update from remote peer 6c70fbdc8a0b42ad8c09c6dbc842cc96: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:53:03.164822 29571 consensus_queue.cc:1035] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:53:03.165271 29573 consensus_queue.cc:1035] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:53:03.165657 29570 consensus_queue.cc:1035] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:53:03.178517 29570 raft_consensus.cc:2932] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6ecf0a12e15f45a3bfa042f88c100a5e (127.25.109.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } }
I20240502 01:53:03.179692 28160 raft_consensus.cc:2932] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6ecf0a12e15f45a3bfa042f88c100a5e (127.25.109.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } }
I20240502 01:53:03.181747 26622 raft_consensus.cc:2932] T 895cfddfe1804d34a80b8a52e00efce2 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6ecf0a12e15f45a3bfa042f88c100a5e (127.25.109.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } }
I20240502 01:53:03.184062 27891 raft_consensus.cc:2932] T 895cfddfe1804d34a80b8a52e00efce2 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6ecf0a12e15f45a3bfa042f88c100a5e (127.25.109.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } }
I20240502 01:53:03.185607 26075 catalog_manager.cc:5484] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 reported cstate change: config changed from index 7 to 8, 6ecf0a12e15f45a3bfa042f88c100a5e (127.25.109.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:03.223382 29583 raft_consensus.cc:1050] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f: attempting to promote NON_VOTER aed397efb8844970b606bca15a77cb59 to VOTER
I20240502 01:53:03.224498 29583 consensus_queue.cc:237] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } }
I20240502 01:53:03.227283 26622 raft_consensus.cc:1261] T 38dbb02fca694854ab66393a87c84908 P d1e7d0eabafa46f382e19f596e2fc00d [term 2 FOLLOWER]: Refusing update from remote peer 868e6e4358f4475bb45967b05eccb09f: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:53:03.227327 27891 raft_consensus.cc:1261] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 2 FOLLOWER]: Refusing update from remote peer 868e6e4358f4475bb45967b05eccb09f: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:53:03.228087 29443 consensus_queue.cc:1035] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:53:03.228544 28026 raft_consensus.cc:1261] T 38dbb02fca694854ab66393a87c84908 P aed397efb8844970b606bca15a77cb59 [term 2 LEARNER]: Refusing update from remote peer 868e6e4358f4475bb45967b05eccb09f: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:53:03.228735 29443 consensus_queue.cc:1035] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:53:03.229233 29443 consensus_queue.cc:1035] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [LEADER]: Connected to new peer: Peer: permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:53:03.245126 29443 raft_consensus.cc:2932] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, aed397efb8844970b606bca15a77cb59 (127.25.109.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } }
I20240502 01:53:03.245911 28026 raft_consensus.cc:2932] T 38dbb02fca694854ab66393a87c84908 P aed397efb8844970b606bca15a77cb59 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, aed397efb8844970b606bca15a77cb59 (127.25.109.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } }
I20240502 01:53:03.246194 27891 raft_consensus.cc:2932] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, aed397efb8844970b606bca15a77cb59 (127.25.109.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } }
I20240502 01:53:03.250893 26076 catalog_manager.cc:5484] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f reported cstate change: config changed from index 8 to 9, aed397efb8844970b606bca15a77cb59 (127.25.109.135) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "868e6e4358f4475bb45967b05eccb09f" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:03.251233 26622 raft_consensus.cc:2932] T 38dbb02fca694854ab66393a87c84908 P d1e7d0eabafa46f382e19f596e2fc00d [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, aed397efb8844970b606bca15a77cb59 (127.25.109.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } }
I20240502 01:53:03.255573 26356 consensus_queue.cc:237] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } }
I20240502 01:53:03.257898 28026 raft_consensus.cc:1261] T 38dbb02fca694854ab66393a87c84908 P aed397efb8844970b606bca15a77cb59 [term 2 FOLLOWER]: Refusing update from remote peer 868e6e4358f4475bb45967b05eccb09f: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:53:03.257933 27891 raft_consensus.cc:1261] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 2 FOLLOWER]: Refusing update from remote peer 868e6e4358f4475bb45967b05eccb09f: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:53:03.258520 29583 consensus_queue.cc:1035] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [LEADER]: Connected to new peer: Peer: permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:53:03.259554 29583 consensus_queue.cc:1035] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:53:03.262723 29584 raft_consensus.cc:2932] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } }
I20240502 01:53:03.263676 28026 raft_consensus.cc:2932] T 38dbb02fca694854ab66393a87c84908 P aed397efb8844970b606bca15a77cb59 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } }
I20240502 01:53:03.265334 27890 raft_consensus.cc:2932] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } }
I20240502 01:53:03.266937 26067 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 38dbb02fca694854ab66393a87c84908 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:03.269322 26078 catalog_manager.cc:5484] T 38dbb02fca694854ab66393a87c84908 P aed397efb8844970b606bca15a77cb59 reported cstate change: config changed from index 9 to 10, VOTER d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132) evicted. New cstate: current_term: 2 leader_uuid: "868e6e4358f4475bb45967b05eccb09f" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } }
I20240502 01:53:03.273447 26601 tablet_service.cc:1514] Processing DeleteTablet for tablet 38dbb02fca694854ab66393a87c84908 with delete_type TABLET_DATA_TOMBSTONED (TS d1e7d0eabafa46f382e19f596e2fc00d not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:59648
I20240502 01:53:03.274567 29591 tablet_replica.cc:330] T 38dbb02fca694854ab66393a87c84908 P d1e7d0eabafa46f382e19f596e2fc00d: stopping tablet replica
I20240502 01:53:03.274763 29591 raft_consensus.cc:2229] T 38dbb02fca694854ab66393a87c84908 P d1e7d0eabafa46f382e19f596e2fc00d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:03.275035 29591 raft_consensus.cc:2258] T 38dbb02fca694854ab66393a87c84908 P d1e7d0eabafa46f382e19f596e2fc00d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:03.275976 29591 ts_tablet_manager.cc:1875] T 38dbb02fca694854ab66393a87c84908 P d1e7d0eabafa46f382e19f596e2fc00d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:03.280892 29591 ts_tablet_manager.cc:1888] T 38dbb02fca694854ab66393a87c84908 P d1e7d0eabafa46f382e19f596e2fc00d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 01:53:03.281085 29591 log.cc:1196] T 38dbb02fca694854ab66393a87c84908 P d1e7d0eabafa46f382e19f596e2fc00d: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal/wals/38dbb02fca694854ab66393a87c84908
I20240502 01:53:03.281965 26067 catalog_manager.cc:4830] TS d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132:37695): tablet 38dbb02fca694854ab66393a87c84908 (table rebalance_test_table_3 [id=3dba071b1adb47bebabd07a94b201ce1]) successfully deleted
I20240502 01:53:03.299067 29573 raft_consensus.cc:1050] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: attempting to promote NON_VOTER ca931e9360d44079b6059b8d9e7a6d20 to VOTER
I20240502 01:53:03.300213 29573 consensus_queue.cc:237] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: false } }
I20240502 01:53:03.304494 28026 raft_consensus.cc:1261] T d4fa5ee9130340f982a6f0bcc0b28b40 P aed397efb8844970b606bca15a77cb59 [term 1 FOLLOWER]: Refusing update from remote peer 6c70fbdc8a0b42ad8c09c6dbc842cc96: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:53:03.304811 26356 raft_consensus.cc:1261] T d4fa5ee9130340f982a6f0bcc0b28b40 P 868e6e4358f4475bb45967b05eccb09f [term 1 FOLLOWER]: Refusing update from remote peer 6c70fbdc8a0b42ad8c09c6dbc842cc96: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:53:03.303843 27757 raft_consensus.cc:1261] T d4fa5ee9130340f982a6f0bcc0b28b40 P ca931e9360d44079b6059b8d9e7a6d20 [term 1 LEARNER]: Refusing update from remote peer 6c70fbdc8a0b42ad8c09c6dbc842cc96: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:53:03.305728 29571 consensus_queue.cc:1035] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:53:03.306156 29577 consensus_queue.cc:1035] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:53:03.306489 29573 consensus_queue.cc:1035] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:53:03.316434 26490 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "895cfddfe1804d34a80b8a52e00efce2"
dest_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:46374
I20240502 01:53:03.316706 26490 raft_consensus.cc:595] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:53:03.319619 29571 raft_consensus.cc:2932] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ca931e9360d44079b6059b8d9e7a6d20 (127.25.109.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: false } } }
I20240502 01:53:03.320946 27757 raft_consensus.cc:2932] T d4fa5ee9130340f982a6f0bcc0b28b40 P ca931e9360d44079b6059b8d9e7a6d20 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ca931e9360d44079b6059b8d9e7a6d20 (127.25.109.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: false } } }
I20240502 01:53:03.326009 26080 catalog_manager.cc:5484] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 reported cstate change: config changed from index 7 to 8, ca931e9360d44079b6059b8d9e7a6d20 (127.25.109.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:03.330404 26356 raft_consensus.cc:2932] T d4fa5ee9130340f982a6f0bcc0b28b40 P 868e6e4358f4475bb45967b05eccb09f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ca931e9360d44079b6059b8d9e7a6d20 (127.25.109.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: false } } }
I20240502 01:53:03.331372 28026 raft_consensus.cc:2932] T d4fa5ee9130340f982a6f0bcc0b28b40 P aed397efb8844970b606bca15a77cb59 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ca931e9360d44079b6059b8d9e7a6d20 (127.25.109.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: false } } }
I20240502 01:53:03.334765 26490 consensus_queue.cc:237] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: false } }
I20240502 01:53:03.336956 28025 raft_consensus.cc:1261] T d4fa5ee9130340f982a6f0bcc0b28b40 P aed397efb8844970b606bca15a77cb59 [term 1 FOLLOWER]: Refusing update from remote peer 6c70fbdc8a0b42ad8c09c6dbc842cc96: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:53:03.337487 29577 consensus_queue.cc:1035] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:53:03.339694 29573 raft_consensus.cc:2932] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 868e6e4358f4475bb45967b05eccb09f (127.25.109.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: false } } }
I20240502 01:53:03.340184 27756 raft_consensus.cc:1261] T d4fa5ee9130340f982a6f0bcc0b28b40 P ca931e9360d44079b6059b8d9e7a6d20 [term 1 FOLLOWER]: Refusing update from remote peer 6c70fbdc8a0b42ad8c09c6dbc842cc96: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:53:03.340811 28025 raft_consensus.cc:2932] T d4fa5ee9130340f982a6f0bcc0b28b40 P aed397efb8844970b606bca15a77cb59 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 868e6e4358f4475bb45967b05eccb09f (127.25.109.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: false } } }
I20240502 01:53:03.341636 29577 consensus_queue.cc:1035] T d4fa5ee9130340f982a6f0bcc0b28b40 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:53:03.342916 27756 raft_consensus.cc:2932] T d4fa5ee9130340f982a6f0bcc0b28b40 P ca931e9360d44079b6059b8d9e7a6d20 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 868e6e4358f4475bb45967b05eccb09f (127.25.109.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: false } } }
I20240502 01:53:03.343417 26068 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d4fa5ee9130340f982a6f0bcc0b28b40 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:03.347016 26076 catalog_manager.cc:5484] T d4fa5ee9130340f982a6f0bcc0b28b40 P ca931e9360d44079b6059b8d9e7a6d20 reported cstate change: config changed from index 8 to 9, VOTER 868e6e4358f4475bb45967b05eccb09f (127.25.109.130) evicted. New cstate: current_term: 1 leader_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: false } } }
I20240502 01:53:03.351233 26335 tablet_service.cc:1514] Processing DeleteTablet for tablet d4fa5ee9130340f982a6f0bcc0b28b40 with delete_type TABLET_DATA_TOMBSTONED (TS 868e6e4358f4475bb45967b05eccb09f not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:47846
I20240502 01:53:03.352175 29614 tablet_replica.cc:330] T d4fa5ee9130340f982a6f0bcc0b28b40 P 868e6e4358f4475bb45967b05eccb09f: stopping tablet replica
I20240502 01:53:03.352419 29614 raft_consensus.cc:2229] T d4fa5ee9130340f982a6f0bcc0b28b40 P 868e6e4358f4475bb45967b05eccb09f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:03.352959 29614 raft_consensus.cc:2258] T d4fa5ee9130340f982a6f0bcc0b28b40 P 868e6e4358f4475bb45967b05eccb09f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:03.354033 29614 ts_tablet_manager.cc:1875] T d4fa5ee9130340f982a6f0bcc0b28b40 P 868e6e4358f4475bb45967b05eccb09f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:03.360306 29614 ts_tablet_manager.cc:1888] T d4fa5ee9130340f982a6f0bcc0b28b40 P 868e6e4358f4475bb45967b05eccb09f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 01:53:03.360536 29614 log.cc:1196] T d4fa5ee9130340f982a6f0bcc0b28b40 P 868e6e4358f4475bb45967b05eccb09f: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal/wals/d4fa5ee9130340f982a6f0bcc0b28b40
I20240502 01:53:03.361574 26067 catalog_manager.cc:4830] TS 868e6e4358f4475bb45967b05eccb09f (127.25.109.130:45571): tablet d4fa5ee9130340f982a6f0bcc0b28b40 (table rebalance_test_table_2 [id=ce4b1d786b674a8fa90b2b56dcb5cd1e]) successfully deleted
I20240502 01:53:03.377768 26490 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "895cfddfe1804d34a80b8a52e00efce2"
dest_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:46402
I20240502 01:53:03.378036 26490 raft_consensus.cc:595] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:53:03.596994 26490 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "895cfddfe1804d34a80b8a52e00efce2"
dest_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:46418
I20240502 01:53:03.597268 26490 raft_consensus.cc:595] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:53:03.604113 29573 raft_consensus.cc:979] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: : Instructing follower c292d733f9584891b46f52aafee403fc to start an election
I20240502 01:53:03.604321 29573 raft_consensus.cc:1067] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 LEADER]: Signalling peer c292d733f9584891b46f52aafee403fc to start an election
I20240502 01:53:03.605129 28160 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "895cfddfe1804d34a80b8a52e00efce2"
dest_uuid: "c292d733f9584891b46f52aafee403fc"
 from {username='slave'} at 127.25.109.131:47861
I20240502 01:53:03.605404 28160 raft_consensus.cc:482] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:53:03.605536 28160 raft_consensus.cc:3037] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:03.607734 28160 raft_consensus.cc:504] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } }
I20240502 01:53:03.609441 28160 leader_election.cc:288] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc [CANDIDATE]: Term 2 election: Requested vote from peers d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132:37695), 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131:38693), 6ecf0a12e15f45a3bfa042f88c100a5e (127.25.109.134:43811)
I20240502 01:53:03.609900 26490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "895cfddfe1804d34a80b8a52e00efce2" candidate_uuid: "c292d733f9584891b46f52aafee403fc" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96"
I20240502 01:53:03.609932 26622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "895cfddfe1804d34a80b8a52e00efce2" candidate_uuid: "c292d733f9584891b46f52aafee403fc" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "d1e7d0eabafa46f382e19f596e2fc00d"
I20240502 01:53:03.610136 26622 raft_consensus.cc:3037] T 895cfddfe1804d34a80b8a52e00efce2 P d1e7d0eabafa46f382e19f596e2fc00d [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:03.610162 26490 raft_consensus.cc:3032] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:53:03.610239 26490 raft_consensus.cc:729] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6c70fbdc8a0b42ad8c09c6dbc842cc96, State: Running, Role: LEADER
I20240502 01:53:03.610581 26490 consensus_queue.cc:260] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } }
I20240502 01:53:03.611024 26490 raft_consensus.cc:3037] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:03.613088 26622 raft_consensus.cc:2454] T 895cfddfe1804d34a80b8a52e00efce2 P d1e7d0eabafa46f382e19f596e2fc00d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c292d733f9584891b46f52aafee403fc in term 2.
I20240502 01:53:03.613708 26490 raft_consensus.cc:2454] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c292d733f9584891b46f52aafee403fc in term 2.
I20240502 01:53:03.614357 28093 leader_election.cc:302] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6c70fbdc8a0b42ad8c09c6dbc842cc96, c292d733f9584891b46f52aafee403fc, d1e7d0eabafa46f382e19f596e2fc00d; no voters: 
I20240502 01:53:03.615144 29579 raft_consensus.cc:2781] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:53:03.615782 29579 raft_consensus.cc:686] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc [term 2 LEADER]: Becoming Leader. State: Replica: c292d733f9584891b46f52aafee403fc, State: Running, Role: LEADER
I20240502 01:53:03.616470 29579 consensus_queue.cc:237] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } }
I20240502 01:53:03.620414 26080 catalog_manager.cc:5484] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc reported cstate change: term changed from 1 to 2, leader changed from 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131) to c292d733f9584891b46f52aafee403fc (127.25.109.136). New cstate: current_term: 2 leader_uuid: "c292d733f9584891b46f52aafee403fc" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:03.621390 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "895cfddfe1804d34a80b8a52e00efce2" candidate_uuid: "c292d733f9584891b46f52aafee403fc" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e"
I20240502 01:53:03.621650 27890 raft_consensus.cc:3037] T 895cfddfe1804d34a80b8a52e00efce2 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:03.623999 27890 raft_consensus.cc:2454] T 895cfddfe1804d34a80b8a52e00efce2 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c292d733f9584891b46f52aafee403fc in term 2.
W20240502 01:53:03.629805 26556 consensus_peers.cc:485] T 4e6854ffbc944e45b51256143505cdb8 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:53:03.679224 26424 consensus_peers.cc:485] T ebe8589feed3400aa6bca73df420778d P 6c70fbdc8a0b42ad8c09c6dbc842cc96 -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:53:03.835721 26292 consensus_peers.cc:485] T fbd456fceb0f4c6db45662eb619cfd21 P 868e6e4358f4475bb45967b05eccb09f -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:53:03.907145 26424 consensus_peers.cc:485] T 64c74d4af3254bc28110d5b6d3b1c51f P 6c70fbdc8a0b42ad8c09c6dbc842cc96 -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:53:03.951488 26556 consensus_peers.cc:485] T e425e061e59a4bffa56b498fdf462a58 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:53:04.015384 26622 raft_consensus.cc:1261] T 895cfddfe1804d34a80b8a52e00efce2 P d1e7d0eabafa46f382e19f596e2fc00d [term 2 FOLLOWER]: Refusing update from remote peer c292d733f9584891b46f52aafee403fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:53:04.016441 29579 consensus_queue.cc:1035] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:53:04.140755 26490 raft_consensus.cc:1261] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 2 FOLLOWER]: Refusing update from remote peer c292d733f9584891b46f52aafee403fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:53:04.141373 29579 consensus_queue.cc:1035] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:53:04.146355 27890 raft_consensus.cc:1261] T 895cfddfe1804d34a80b8a52e00efce2 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 2 FOLLOWER]: Refusing update from remote peer c292d733f9584891b46f52aafee403fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:53:04.147543 29579 consensus_queue.cc:1035] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 01:53:04.163327 26292 consensus_peers.cc:485] T 7def725255ba4448837b47ba65a6ed44 P 868e6e4358f4475bb45967b05eccb09f -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:53:04.520928 26556 consensus_peers.cc:485] T 01b7ac587d134a61b333489d937c0a21 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:53:04.609088 28160 consensus_queue.cc:237] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } }
I20240502 01:53:04.611016 27890 raft_consensus.cc:1261] T 895cfddfe1804d34a80b8a52e00efce2 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 2 FOLLOWER]: Refusing update from remote peer c292d733f9584891b46f52aafee403fc: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:53:04.611037 26622 raft_consensus.cc:1261] T 895cfddfe1804d34a80b8a52e00efce2 P d1e7d0eabafa46f382e19f596e2fc00d [term 2 FOLLOWER]: Refusing update from remote peer c292d733f9584891b46f52aafee403fc: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:53:04.611624 29579 consensus_queue.cc:1035] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:53:04.612039 29654 consensus_queue.cc:1035] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:53:04.614297 29579 raft_consensus.cc:2932] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } }
I20240502 01:53:04.614954 27890 raft_consensus.cc:2932] T 895cfddfe1804d34a80b8a52e00efce2 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } }
I20240502 01:53:04.615213 26622 raft_consensus.cc:2932] T 895cfddfe1804d34a80b8a52e00efce2 P d1e7d0eabafa46f382e19f596e2fc00d [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } }
I20240502 01:53:04.619505 26066 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 895cfddfe1804d34a80b8a52e00efce2 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:04.619782 26080 catalog_manager.cc:5484] T 895cfddfe1804d34a80b8a52e00efce2 P d1e7d0eabafa46f382e19f596e2fc00d reported cstate change: config changed from index 8 to 10, VOTER 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131) evicted. New cstate: current_term: 2 leader_uuid: "c292d733f9584891b46f52aafee403fc" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } }
I20240502 01:53:04.624167 26467 tablet_service.cc:1514] Processing DeleteTablet for tablet 895cfddfe1804d34a80b8a52e00efce2 with delete_type TABLET_DATA_TOMBSTONED (TS 6c70fbdc8a0b42ad8c09c6dbc842cc96 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:39030
I20240502 01:53:04.625000 29676 tablet_replica.cc:330] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: stopping tablet replica
I20240502 01:53:04.625185 29676 raft_consensus.cc:2229] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:04.625417 29676 raft_consensus.cc:2258] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:04.626156 29676 ts_tablet_manager.cc:1875] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:04.630581 29676 ts_tablet_manager.cc:1888] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 01:53:04.630697 29676 log.cc:1196] T 895cfddfe1804d34a80b8a52e00efce2 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/wals/895cfddfe1804d34a80b8a52e00efce2
I20240502 01:53:04.631378 26068 catalog_manager.cc:4830] TS 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131:38693): tablet 895cfddfe1804d34a80b8a52e00efce2 (table rebalance_test_table_0 [id=45682f0e9b4f45c3b3e4cb8de40b23bd]) successfully deleted
I20240502 01:53:04.669821 27890 consensus_queue.cc:237] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: true } }
I20240502 01:53:04.673256 28160 raft_consensus.cc:1261] T 7aa623407e7844989e7bf34db5d6ba35 P c292d733f9584891b46f52aafee403fc [term 2 FOLLOWER]: Refusing update from remote peer 6ecf0a12e15f45a3bfa042f88c100a5e: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 01:53:04.674266 29659 consensus_queue.cc:1035] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:53:04.679327 26490 raft_consensus.cc:1261] T 7aa623407e7844989e7bf34db5d6ba35 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 2 FOLLOWER]: Refusing update from remote peer 6ecf0a12e15f45a3bfa042f88c100a5e: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 01:53:04.680374 29659 consensus_queue.cc:1035] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:53:04.686041 29659 raft_consensus.cc:2932] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER ca931e9360d44079b6059b8d9e7a6d20 (127.25.109.133) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: true } } }
I20240502 01:53:04.686996 26490 raft_consensus.cc:2932] T 7aa623407e7844989e7bf34db5d6ba35 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER ca931e9360d44079b6059b8d9e7a6d20 (127.25.109.133) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: true } } }
I20240502 01:53:04.693344 26078 catalog_manager.cc:5484] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e reported cstate change: config changed from index 11 to 12, NON_VOTER ca931e9360d44079b6059b8d9e7a6d20 (127.25.109.133) added. New cstate: current_term: 2 leader_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:53:04.700382 27825 consensus_peers.cc:485] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e -> Peer ca931e9360d44079b6059b8d9e7a6d20 (127.25.109.133:33081): Couldn't send request to peer ca931e9360d44079b6059b8d9e7a6d20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7aa623407e7844989e7bf34db5d6ba35. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:04.705735 28160 raft_consensus.cc:2932] T 7aa623407e7844989e7bf34db5d6ba35 P c292d733f9584891b46f52aafee403fc [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER ca931e9360d44079b6059b8d9e7a6d20 (127.25.109.133) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: true } } }
W20240502 01:53:04.818750 26556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 01:53:04.818792 26292 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240502 01:53:04.864096 26292 consensus_peers.cc:485] T 062fa1dfab1046f287e2ddb1c240c4eb P 868e6e4358f4475bb45967b05eccb09f -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:53:04.992142 26556 consensus_peers.cc:485] T 536039b70452486684f7075cf45c1851 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:53:05.089933 29706 ts_tablet_manager.cc:902] T 7aa623407e7844989e7bf34db5d6ba35 P ca931e9360d44079b6059b8d9e7a6d20: Initiating tablet copy from peer 6ecf0a12e15f45a3bfa042f88c100a5e (127.25.109.134:43811)
I20240502 01:53:05.091281 29706 tablet_copy_client.cc:308] T 7aa623407e7844989e7bf34db5d6ba35 P ca931e9360d44079b6059b8d9e7a6d20: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.134:43811
I20240502 01:53:05.097658 27911 tablet_copy_service.cc:140] P 6ecf0a12e15f45a3bfa042f88c100a5e: Received BeginTabletCopySession request for tablet 7aa623407e7844989e7bf34db5d6ba35 from peer ca931e9360d44079b6059b8d9e7a6d20 ({username='slave'} at 127.25.109.133:57603)
I20240502 01:53:05.098011 27911 tablet_copy_service.cc:161] P 6ecf0a12e15f45a3bfa042f88c100a5e: Beginning new tablet copy session on tablet 7aa623407e7844989e7bf34db5d6ba35 from peer ca931e9360d44079b6059b8d9e7a6d20 at {username='slave'} at 127.25.109.133:57603: session id = ca931e9360d44079b6059b8d9e7a6d20-7aa623407e7844989e7bf34db5d6ba35
I20240502 01:53:05.100845 27911 tablet_copy_source_session.cc:205] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:05.102537 29706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aa623407e7844989e7bf34db5d6ba35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:05.107257 29706 tablet_copy_client.cc:783] T 7aa623407e7844989e7bf34db5d6ba35 P ca931e9360d44079b6059b8d9e7a6d20: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:05.107535 29706 tablet_copy_client.cc:647] T 7aa623407e7844989e7bf34db5d6ba35 P ca931e9360d44079b6059b8d9e7a6d20: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:05.109431 29710 tablet_copy_client.cc:1002] T 7aa623407e7844989e7bf34db5d6ba35 P ca931e9360d44079b6059b8d9e7a6d20: tablet copy: Injecting latency into file download: 1500
W20240502 01:53:05.206535 26556 consensus_peers.cc:485] T dc44aad1f4ab4469a29d280a11268be0 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:53:05.284581 26292 consensus_peers.cc:485] T c32bdf0d644e499eaa2f755fab9abb77 P 868e6e4358f4475bb45967b05eccb09f -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:53:06.115990 26424 consensus_peers.cc:485] T ebe8589feed3400aa6bca73df420778d P 6c70fbdc8a0b42ad8c09c6dbc842cc96 -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:53:06.282464 26424 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 01:53:06.283514 26424 consensus_peers.cc:485] T 64c74d4af3254bc28110d5b6d3b1c51f P 6c70fbdc8a0b42ad8c09c6dbc842cc96 -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:53:06.297407 26556 consensus_peers.cc:485] T 4e6854ffbc944e45b51256143505cdb8 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:53:06.351161 26292 consensus_peers.cc:485] T 7def725255ba4448837b47ba65a6ed44 P 868e6e4358f4475bb45967b05eccb09f -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:53:06.453753 26292 consensus_peers.cc:485] T fbd456fceb0f4c6db45662eb619cfd21 P 868e6e4358f4475bb45967b05eccb09f -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:53:06.503978 26556 consensus_peers.cc:485] T e425e061e59a4bffa56b498fdf462a58 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:53:06.609912 29706 tablet_copy_client.cc:523] T 7aa623407e7844989e7bf34db5d6ba35 P ca931e9360d44079b6059b8d9e7a6d20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:06.613519 29706 tablet_bootstrap.cc:492] T 7aa623407e7844989e7bf34db5d6ba35 P ca931e9360d44079b6059b8d9e7a6d20: Bootstrap starting.
I20240502 01:53:06.649327 29706 tablet_bootstrap.cc:492] T 7aa623407e7844989e7bf34db5d6ba35 P ca931e9360d44079b6059b8d9e7a6d20: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:06.649789 29706 tablet_bootstrap.cc:492] T 7aa623407e7844989e7bf34db5d6ba35 P ca931e9360d44079b6059b8d9e7a6d20: Bootstrap complete.
I20240502 01:53:06.650060 29706 ts_tablet_manager.cc:1367] T 7aa623407e7844989e7bf34db5d6ba35 P ca931e9360d44079b6059b8d9e7a6d20: Time spent bootstrapping tablet: real 0.037s	user 0.036s	sys 0.000s
I20240502 01:53:06.650925 29706 raft_consensus.cc:348] T 7aa623407e7844989e7bf34db5d6ba35 P ca931e9360d44079b6059b8d9e7a6d20 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: true } }
I20240502 01:53:06.651165 29706 raft_consensus.cc:729] T 7aa623407e7844989e7bf34db5d6ba35 P ca931e9360d44079b6059b8d9e7a6d20 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: ca931e9360d44079b6059b8d9e7a6d20, State: Initialized, Role: LEARNER
I20240502 01:53:06.651371 29706 consensus_queue.cc:260] T 7aa623407e7844989e7bf34db5d6ba35 P ca931e9360d44079b6059b8d9e7a6d20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: true } }
I20240502 01:53:06.652144 29706 ts_tablet_manager.cc:1398] T 7aa623407e7844989e7bf34db5d6ba35 P ca931e9360d44079b6059b8d9e7a6d20: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240502 01:53:06.653045 27911 tablet_copy_service.cc:342] P 6ecf0a12e15f45a3bfa042f88c100a5e: Request end of tablet copy session ca931e9360d44079b6059b8d9e7a6d20-7aa623407e7844989e7bf34db5d6ba35 received from {username='slave'} at 127.25.109.133:57603
I20240502 01:53:06.653218 27911 tablet_copy_service.cc:433] P 6ecf0a12e15f45a3bfa042f88c100a5e: ending tablet copy session ca931e9360d44079b6059b8d9e7a6d20-7aa623407e7844989e7bf34db5d6ba35 on tablet 7aa623407e7844989e7bf34db5d6ba35 with peer ca931e9360d44079b6059b8d9e7a6d20
I20240502 01:53:07.008538 27756 raft_consensus.cc:1203] T 7aa623407e7844989e7bf34db5d6ba35 P ca931e9360d44079b6059b8d9e7a6d20 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12]   Dedup: 2.12->[]
W20240502 01:53:07.068523 26556 consensus_peers.cc:485] T 01b7ac587d134a61b333489d937c0a21 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:53:07.285311 26292 consensus_peers.cc:485] T 062fa1dfab1046f287e2ddb1c240c4eb P 868e6e4358f4475bb45967b05eccb09f -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:53:07.620479 29779 raft_consensus.cc:1050] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e: attempting to promote NON_VOTER ca931e9360d44079b6059b8d9e7a6d20 to VOTER
I20240502 01:53:07.621316 29779 consensus_queue.cc:237] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: false } }
I20240502 01:53:07.623754 27756 raft_consensus.cc:1261] T 7aa623407e7844989e7bf34db5d6ba35 P ca931e9360d44079b6059b8d9e7a6d20 [term 2 LEARNER]: Refusing update from remote peer 6ecf0a12e15f45a3bfa042f88c100a5e: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 01:53:07.623831 26490 raft_consensus.cc:1261] T 7aa623407e7844989e7bf34db5d6ba35 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 2 FOLLOWER]: Refusing update from remote peer 6ecf0a12e15f45a3bfa042f88c100a5e: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 01:53:07.624054 28160 raft_consensus.cc:1261] T 7aa623407e7844989e7bf34db5d6ba35 P c292d733f9584891b46f52aafee403fc [term 2 FOLLOWER]: Refusing update from remote peer 6ecf0a12e15f45a3bfa042f88c100a5e: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 01:53:07.624547 29779 consensus_queue.cc:1035] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:53:07.624935 29781 consensus_queue.cc:1035] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:53:07.625293 29778 consensus_queue.cc:1035] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:53:07.628335 29778 raft_consensus.cc:2932] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, ca931e9360d44079b6059b8d9e7a6d20 (127.25.109.133) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: false } } }
I20240502 01:53:07.629006 26490 raft_consensus.cc:2932] T 7aa623407e7844989e7bf34db5d6ba35 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, ca931e9360d44079b6059b8d9e7a6d20 (127.25.109.133) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: false } } }
I20240502 01:53:07.632571 28160 raft_consensus.cc:2932] T 7aa623407e7844989e7bf34db5d6ba35 P c292d733f9584891b46f52aafee403fc [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, ca931e9360d44079b6059b8d9e7a6d20 (127.25.109.133) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: false } } }
I20240502 01:53:07.632936 26076 catalog_manager.cc:5484] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e reported cstate change: config changed from index 12 to 13, ca931e9360d44079b6059b8d9e7a6d20 (127.25.109.133) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:07.633121 27756 raft_consensus.cc:2932] T 7aa623407e7844989e7bf34db5d6ba35 P ca931e9360d44079b6059b8d9e7a6d20 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, ca931e9360d44079b6059b8d9e7a6d20 (127.25.109.133) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: false } } }
I20240502 01:53:07.640694 27890 consensus_queue.cc:237] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: false } }
I20240502 01:53:07.642578 27756 raft_consensus.cc:1261] T 7aa623407e7844989e7bf34db5d6ba35 P ca931e9360d44079b6059b8d9e7a6d20 [term 2 FOLLOWER]: Refusing update from remote peer 6ecf0a12e15f45a3bfa042f88c100a5e: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240502 01:53:07.642625 28160 raft_consensus.cc:1261] T 7aa623407e7844989e7bf34db5d6ba35 P c292d733f9584891b46f52aafee403fc [term 2 FOLLOWER]: Refusing update from remote peer 6ecf0a12e15f45a3bfa042f88c100a5e: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240502 01:53:07.643074 29727 consensus_queue.cc:1035] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 01:53:07.643389 29779 consensus_queue.cc:1035] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 01:53:07.645517 29778 raft_consensus.cc:2932] T 7aa623407e7844989e7bf34db5d6ba35 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: false } } }
I20240502 01:53:07.646143 27756 raft_consensus.cc:2932] T 7aa623407e7844989e7bf34db5d6ba35 P ca931e9360d44079b6059b8d9e7a6d20 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: false } } }
I20240502 01:53:07.646157 28160 raft_consensus.cc:2932] T 7aa623407e7844989e7bf34db5d6ba35 P c292d733f9584891b46f52aafee403fc [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: false } } }
I20240502 01:53:07.649387 26066 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7aa623407e7844989e7bf34db5d6ba35 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240502 01:53:07.651823 26556 consensus_peers.cc:485] T 536039b70452486684f7075cf45c1851 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:53:07.651743 26080 catalog_manager.cc:5484] T 7aa623407e7844989e7bf34db5d6ba35 P ca931e9360d44079b6059b8d9e7a6d20 reported cstate change: config changed from index 13 to 14, VOTER 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131) evicted. New cstate: current_term: 2 leader_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: false } } }
I20240502 01:53:07.655624 26467 tablet_service.cc:1514] Processing DeleteTablet for tablet 7aa623407e7844989e7bf34db5d6ba35 with delete_type TABLET_DATA_TOMBSTONED (TS 6c70fbdc8a0b42ad8c09c6dbc842cc96 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:39030
I20240502 01:53:07.656448 29790 tablet_replica.cc:330] T 7aa623407e7844989e7bf34db5d6ba35 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: stopping tablet replica
I20240502 01:53:07.656682 29790 raft_consensus.cc:2229] T 7aa623407e7844989e7bf34db5d6ba35 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:07.657243 29790 raft_consensus.cc:2258] T 7aa623407e7844989e7bf34db5d6ba35 P 6c70fbdc8a0b42ad8c09c6dbc842cc96 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:07.658124 29790 ts_tablet_manager.cc:1875] T 7aa623407e7844989e7bf34db5d6ba35 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:07.663376 29790 ts_tablet_manager.cc:1888] T 7aa623407e7844989e7bf34db5d6ba35 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.13
I20240502 01:53:07.663511 29790 log.cc:1196] T 7aa623407e7844989e7bf34db5d6ba35 P 6c70fbdc8a0b42ad8c09c6dbc842cc96: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/wals/7aa623407e7844989e7bf34db5d6ba35
I20240502 01:53:07.664316 26068 catalog_manager.cc:4830] TS 6c70fbdc8a0b42ad8c09c6dbc842cc96 (127.25.109.131:38693): tablet 7aa623407e7844989e7bf34db5d6ba35 (table rebalance_test_table_4 [id=690e246e360e425598dedef6d62ce625]) successfully deleted
W20240502 01:53:07.717708 26292 consensus_peers.cc:485] T c32bdf0d644e499eaa2f755fab9abb77 P 868e6e4358f4475bb45967b05eccb09f -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:53:07.803347 26467 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 01:53:07.816007 26556 consensus_peers.cc:485] T dc44aad1f4ab4469a29d280a11268be0 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:53:07.831939 26068 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111)
I20240502 01:53:07.833248 27869 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:53:07.860648 26068 catalog_manager.cc:4573] TS f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 7aa623407e7844989e7bf34db5d6ba35: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111)
I20240502 01:53:07.872047 28139 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:07.874354 26335 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:53:07.888083 28005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:07.937101 27736 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:07.938369 26601 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:53:08.011495 26356 consensus_queue.cc:237] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: true } }
I20240502 01:53:08.014236 27890 raft_consensus.cc:1261] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 2 FOLLOWER]: Refusing update from remote peer 868e6e4358f4475bb45967b05eccb09f: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 01:53:08.014266 28025 raft_consensus.cc:1261] T 38dbb02fca694854ab66393a87c84908 P aed397efb8844970b606bca15a77cb59 [term 2 FOLLOWER]: Refusing update from remote peer 868e6e4358f4475bb45967b05eccb09f: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 01:53:08.014950 29844 consensus_queue.cc:1035] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [LEADER]: Connected to new peer: Peer: permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:53:08.015333 29586 consensus_queue.cc:1035] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:53:08.018896 29844 raft_consensus.cc:2932] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER ca931e9360d44079b6059b8d9e7a6d20 (127.25.109.133) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: true } } }
I20240502 01:53:08.019342 27890 raft_consensus.cc:2932] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER ca931e9360d44079b6059b8d9e7a6d20 (127.25.109.133) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: true } } }
I20240502 01:53:08.020354 28025 raft_consensus.cc:2932] T 38dbb02fca694854ab66393a87c84908 P aed397efb8844970b606bca15a77cb59 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER ca931e9360d44079b6059b8d9e7a6d20 (127.25.109.133) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: true } } }
W20240502 01:53:08.023258 26292 consensus_peers.cc:485] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f -> Peer ca931e9360d44079b6059b8d9e7a6d20 (127.25.109.133:33081): Couldn't send request to peer ca931e9360d44079b6059b8d9e7a6d20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 38dbb02fca694854ab66393a87c84908. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:08.025312 26078 catalog_manager.cc:5484] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e reported cstate change: config changed from index 10 to 11, NON_VOTER ca931e9360d44079b6059b8d9e7a6d20 (127.25.109.133) added. New cstate: current_term: 2 leader_uuid: "868e6e4358f4475bb45967b05eccb09f" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: true } } }
I20240502 01:53:08.039989 28160 consensus_queue.cc:237] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: NON_VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: true } }
I20240502 01:53:08.043119 26622 raft_consensus.cc:1261] T 895cfddfe1804d34a80b8a52e00efce2 P d1e7d0eabafa46f382e19f596e2fc00d [term 2 FOLLOWER]: Refusing update from remote peer c292d733f9584891b46f52aafee403fc: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 01:53:08.043169 27890 raft_consensus.cc:1261] T 895cfddfe1804d34a80b8a52e00efce2 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 2 FOLLOWER]: Refusing update from remote peer c292d733f9584891b46f52aafee403fc: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 01:53:08.043772 29855 consensus_queue.cc:1035] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:53:08.044301 29854 consensus_queue.cc:1035] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:53:08.047394 29855 raft_consensus.cc:2932] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER aed397efb8844970b606bca15a77cb59 (127.25.109.135) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: NON_VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: true } } }
I20240502 01:53:08.048204 27890 raft_consensus.cc:2932] T 895cfddfe1804d34a80b8a52e00efce2 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER aed397efb8844970b606bca15a77cb59 (127.25.109.135) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: NON_VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: true } } }
W20240502 01:53:08.051050 28091 consensus_peers.cc:485] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc -> Peer aed397efb8844970b606bca15a77cb59 (127.25.109.135:39905): Couldn't send request to peer aed397efb8844970b606bca15a77cb59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 895cfddfe1804d34a80b8a52e00efce2. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:08.053426 26080 catalog_manager.cc:5484] T 895cfddfe1804d34a80b8a52e00efce2 P 6ecf0a12e15f45a3bfa042f88c100a5e reported cstate change: config changed from index 10 to 11, NON_VOTER aed397efb8844970b606bca15a77cb59 (127.25.109.135) added. New cstate: current_term: 2 leader_uuid: "c292d733f9584891b46f52aafee403fc" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: NON_VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: true } } }
I20240502 01:53:08.066720 26622 raft_consensus.cc:2932] T 895cfddfe1804d34a80b8a52e00efce2 P d1e7d0eabafa46f382e19f596e2fc00d [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER aed397efb8844970b606bca15a77cb59 (127.25.109.135) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: NON_VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: true } } }
W20240502 01:53:08.617516 26424 consensus_peers.cc:485] T 64c74d4af3254bc28110d5b6d3b1c51f P 6c70fbdc8a0b42ad8c09c6dbc842cc96 -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:53:08.619894 29893 ts_tablet_manager.cc:902] T 38dbb02fca694854ab66393a87c84908 P ca931e9360d44079b6059b8d9e7a6d20: Initiating tablet copy from peer 868e6e4358f4475bb45967b05eccb09f (127.25.109.130:45571)
I20240502 01:53:08.621088 29893 tablet_copy_client.cc:308] T 38dbb02fca694854ab66393a87c84908 P ca931e9360d44079b6059b8d9e7a6d20: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.130:45571
I20240502 01:53:08.624544 29899 ts_tablet_manager.cc:902] T 895cfddfe1804d34a80b8a52e00efce2 P aed397efb8844970b606bca15a77cb59: Initiating tablet copy from peer c292d733f9584891b46f52aafee403fc (127.25.109.136:38319)
I20240502 01:53:08.625952 29899 tablet_copy_client.cc:308] T 895cfddfe1804d34a80b8a52e00efce2 P aed397efb8844970b606bca15a77cb59: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.136:38319
I20240502 01:53:08.627339 26377 tablet_copy_service.cc:140] P 868e6e4358f4475bb45967b05eccb09f: Received BeginTabletCopySession request for tablet 38dbb02fca694854ab66393a87c84908 from peer ca931e9360d44079b6059b8d9e7a6d20 ({username='slave'} at 127.25.109.133:53363)
I20240502 01:53:08.627522 26377 tablet_copy_service.cc:161] P 868e6e4358f4475bb45967b05eccb09f: Beginning new tablet copy session on tablet 38dbb02fca694854ab66393a87c84908 from peer ca931e9360d44079b6059b8d9e7a6d20 at {username='slave'} at 127.25.109.133:53363: session id = ca931e9360d44079b6059b8d9e7a6d20-38dbb02fca694854ab66393a87c84908
I20240502 01:53:08.630585 26377 tablet_copy_source_session.cc:205] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:08.631964 29893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38dbb02fca694854ab66393a87c84908. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:08.633138 28181 tablet_copy_service.cc:140] P c292d733f9584891b46f52aafee403fc: Received BeginTabletCopySession request for tablet 895cfddfe1804d34a80b8a52e00efce2 from peer aed397efb8844970b606bca15a77cb59 ({username='slave'} at 127.25.109.135:39079)
I20240502 01:53:08.633364 28181 tablet_copy_service.cc:161] P c292d733f9584891b46f52aafee403fc: Beginning new tablet copy session on tablet 895cfddfe1804d34a80b8a52e00efce2 from peer aed397efb8844970b606bca15a77cb59 at {username='slave'} at 127.25.109.135:39079: session id = aed397efb8844970b606bca15a77cb59-895cfddfe1804d34a80b8a52e00efce2
I20240502 01:53:08.637027 28181 tablet_copy_source_session.cc:205] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:08.638780 29899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 895cfddfe1804d34a80b8a52e00efce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:08.638221 29893 tablet_copy_client.cc:783] T 38dbb02fca694854ab66393a87c84908 P ca931e9360d44079b6059b8d9e7a6d20: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:08.640153 29893 tablet_copy_client.cc:647] T 38dbb02fca694854ab66393a87c84908 P ca931e9360d44079b6059b8d9e7a6d20: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:08.642398 29897 tablet_copy_client.cc:1002] T 38dbb02fca694854ab66393a87c84908 P ca931e9360d44079b6059b8d9e7a6d20: tablet copy: Injecting latency into file download: 1500
I20240502 01:53:08.644874 29899 tablet_copy_client.cc:783] T 895cfddfe1804d34a80b8a52e00efce2 P aed397efb8844970b606bca15a77cb59: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:08.645112 29899 tablet_copy_client.cc:647] T 895cfddfe1804d34a80b8a52e00efce2 P aed397efb8844970b606bca15a77cb59: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:08.647404 29903 tablet_copy_client.cc:1002] T 895cfddfe1804d34a80b8a52e00efce2 P aed397efb8844970b606bca15a77cb59: tablet copy: Injecting latency into file download: 1500
W20240502 01:53:08.656765 26424 consensus_peers.cc:485] T ebe8589feed3400aa6bca73df420778d P 6c70fbdc8a0b42ad8c09c6dbc842cc96 -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:53:08.741423 26292 consensus_peers.cc:485] T 7def725255ba4448837b47ba65a6ed44 P 868e6e4358f4475bb45967b05eccb09f -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:53:08.766709 26556 consensus_peers.cc:485] T e425e061e59a4bffa56b498fdf462a58 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:53:08.864415 26556 consensus_peers.cc:485] T 4e6854ffbc944e45b51256143505cdb8 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:53:09.056618 26292 consensus_peers.cc:485] T fbd456fceb0f4c6db45662eb619cfd21 P 868e6e4358f4475bb45967b05eccb09f -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:53:09.364542 26556 consensus_peers.cc:485] T 01b7ac587d134a61b333489d937c0a21 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:53:09.776060 26292 consensus_peers.cc:485] T 062fa1dfab1046f287e2ddb1c240c4eb P 868e6e4358f4475bb45967b05eccb09f -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:53:09.823611 26292 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 01:53:10.141531 26556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111) [suppressed 52 similar messages]
I20240502 01:53:10.142781 29893 tablet_copy_client.cc:523] T 38dbb02fca694854ab66393a87c84908 P ca931e9360d44079b6059b8d9e7a6d20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:10.145388 29893 tablet_bootstrap.cc:492] T 38dbb02fca694854ab66393a87c84908 P ca931e9360d44079b6059b8d9e7a6d20: Bootstrap starting.
I20240502 01:53:10.147877 29899 tablet_copy_client.cc:523] T 895cfddfe1804d34a80b8a52e00efce2 P aed397efb8844970b606bca15a77cb59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:10.150596 29899 tablet_bootstrap.cc:492] T 895cfddfe1804d34a80b8a52e00efce2 P aed397efb8844970b606bca15a77cb59: Bootstrap starting.
I20240502 01:53:10.182080 29893 tablet_bootstrap.cc:492] T 38dbb02fca694854ab66393a87c84908 P ca931e9360d44079b6059b8d9e7a6d20: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:10.182628 29893 tablet_bootstrap.cc:492] T 38dbb02fca694854ab66393a87c84908 P ca931e9360d44079b6059b8d9e7a6d20: Bootstrap complete.
I20240502 01:53:10.182947 29893 ts_tablet_manager.cc:1367] T 38dbb02fca694854ab66393a87c84908 P ca931e9360d44079b6059b8d9e7a6d20: Time spent bootstrapping tablet: real 0.038s	user 0.034s	sys 0.001s
I20240502 01:53:10.183823 29893 raft_consensus.cc:348] T 38dbb02fca694854ab66393a87c84908 P ca931e9360d44079b6059b8d9e7a6d20 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: true } }
I20240502 01:53:10.184206 29893 raft_consensus.cc:729] T 38dbb02fca694854ab66393a87c84908 P ca931e9360d44079b6059b8d9e7a6d20 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: ca931e9360d44079b6059b8d9e7a6d20, State: Initialized, Role: LEARNER
I20240502 01:53:10.184448 29893 consensus_queue.cc:260] T 38dbb02fca694854ab66393a87c84908 P ca931e9360d44079b6059b8d9e7a6d20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: true } }
I20240502 01:53:10.185379 29893 ts_tablet_manager.cc:1398] T 38dbb02fca694854ab66393a87c84908 P ca931e9360d44079b6059b8d9e7a6d20: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.004s
I20240502 01:53:10.186371 26377 tablet_copy_service.cc:342] P 868e6e4358f4475bb45967b05eccb09f: Request end of tablet copy session ca931e9360d44079b6059b8d9e7a6d20-38dbb02fca694854ab66393a87c84908 received from {username='slave'} at 127.25.109.133:53363
I20240502 01:53:10.186494 26377 tablet_copy_service.cc:433] P 868e6e4358f4475bb45967b05eccb09f: ending tablet copy session ca931e9360d44079b6059b8d9e7a6d20-38dbb02fca694854ab66393a87c84908 on tablet 38dbb02fca694854ab66393a87c84908 with peer ca931e9360d44079b6059b8d9e7a6d20
I20240502 01:53:10.190312 29899 tablet_bootstrap.cc:492] T 895cfddfe1804d34a80b8a52e00efce2 P aed397efb8844970b606bca15a77cb59: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:10.190790 29899 tablet_bootstrap.cc:492] T 895cfddfe1804d34a80b8a52e00efce2 P aed397efb8844970b606bca15a77cb59: Bootstrap complete.
I20240502 01:53:10.191087 29899 ts_tablet_manager.cc:1367] T 895cfddfe1804d34a80b8a52e00efce2 P aed397efb8844970b606bca15a77cb59: Time spent bootstrapping tablet: real 0.041s	user 0.035s	sys 0.004s
W20240502 01:53:10.191850 26292 consensus_peers.cc:485] T c32bdf0d644e499eaa2f755fab9abb77 P 868e6e4358f4475bb45967b05eccb09f -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 01:53:10.191970 29899 raft_consensus.cc:348] T 895cfddfe1804d34a80b8a52e00efce2 P aed397efb8844970b606bca15a77cb59 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: NON_VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: true } }
I20240502 01:53:10.192248 29899 raft_consensus.cc:729] T 895cfddfe1804d34a80b8a52e00efce2 P aed397efb8844970b606bca15a77cb59 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: aed397efb8844970b606bca15a77cb59, State: Initialized, Role: LEARNER
I20240502 01:53:10.192459 29899 consensus_queue.cc:260] T 895cfddfe1804d34a80b8a52e00efce2 P aed397efb8844970b606bca15a77cb59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: NON_VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: true } }
I20240502 01:53:10.193344 29899 ts_tablet_manager.cc:1398] T 895cfddfe1804d34a80b8a52e00efce2 P aed397efb8844970b606bca15a77cb59: Time spent starting tablet: real 0.002s	user 0.005s	sys 0.000s
I20240502 01:53:10.194157 28181 tablet_copy_service.cc:342] P c292d733f9584891b46f52aafee403fc: Request end of tablet copy session aed397efb8844970b606bca15a77cb59-895cfddfe1804d34a80b8a52e00efce2 received from {username='slave'} at 127.25.109.135:39079
I20240502 01:53:10.194319 28181 tablet_copy_service.cc:433] P c292d733f9584891b46f52aafee403fc: ending tablet copy session aed397efb8844970b606bca15a77cb59-895cfddfe1804d34a80b8a52e00efce2 on tablet 895cfddfe1804d34a80b8a52e00efce2 with peer aed397efb8844970b606bca15a77cb59
W20240502 01:53:10.252648 26556 consensus_peers.cc:485] T 536039b70452486684f7075cf45c1851 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 01:53:10.283475 27756 raft_consensus.cc:1203] T 38dbb02fca694854ab66393a87c84908 P ca931e9360d44079b6059b8d9e7a6d20 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11]   Dedup: 2.11->[]
W20240502 01:53:10.300587 26556 consensus_peers.cc:485] T dc44aad1f4ab4469a29d280a11268be0 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 01:53:10.502315 28025 raft_consensus.cc:1203] T 895cfddfe1804d34a80b8a52e00efce2 P aed397efb8844970b606bca15a77cb59 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11]   Dedup: 2.11->[]
I20240502 01:53:10.908141 30024 raft_consensus.cc:1050] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f: attempting to promote NON_VOTER ca931e9360d44079b6059b8d9e7a6d20 to VOTER
I20240502 01:53:10.908890 30024 consensus_queue.cc:237] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: false } }
I20240502 01:53:10.911732 28025 raft_consensus.cc:1261] T 38dbb02fca694854ab66393a87c84908 P aed397efb8844970b606bca15a77cb59 [term 2 FOLLOWER]: Refusing update from remote peer 868e6e4358f4475bb45967b05eccb09f: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 01:53:10.911919 27756 raft_consensus.cc:1261] T 38dbb02fca694854ab66393a87c84908 P ca931e9360d44079b6059b8d9e7a6d20 [term 2 LEARNER]: Refusing update from remote peer 868e6e4358f4475bb45967b05eccb09f: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 01:53:10.912746 30024 consensus_queue.cc:1035] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [LEADER]: Connected to new peer: Peer: permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:53:10.913148 29967 consensus_queue.cc:1035] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:53:10.913542 27890 raft_consensus.cc:1261] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 2 FOLLOWER]: Refusing update from remote peer 868e6e4358f4475bb45967b05eccb09f: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 01:53:10.914129 29967 consensus_queue.cc:1035] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:53:10.917171 30026 raft_consensus.cc:2932] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, ca931e9360d44079b6059b8d9e7a6d20 (127.25.109.133) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: false } } }
I20240502 01:53:10.919113 28025 raft_consensus.cc:2932] T 38dbb02fca694854ab66393a87c84908 P aed397efb8844970b606bca15a77cb59 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, ca931e9360d44079b6059b8d9e7a6d20 (127.25.109.133) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: false } } }
I20240502 01:53:10.920641 27890 raft_consensus.cc:2932] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, ca931e9360d44079b6059b8d9e7a6d20 (127.25.109.133) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: false } } }
I20240502 01:53:10.923403 26075 catalog_manager.cc:5484] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f reported cstate change: config changed from index 11 to 12, ca931e9360d44079b6059b8d9e7a6d20 (127.25.109.133) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "868e6e4358f4475bb45967b05eccb09f" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:10.924041 27756 raft_consensus.cc:2932] T 38dbb02fca694854ab66393a87c84908 P ca931e9360d44079b6059b8d9e7a6d20 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, ca931e9360d44079b6059b8d9e7a6d20 (127.25.109.133) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: false } } }
I20240502 01:53:10.979984 26357 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "38dbb02fca694854ab66393a87c84908"
dest_uuid: "868e6e4358f4475bb45967b05eccb09f"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:47566
I20240502 01:53:10.980305 26357 raft_consensus.cc:595] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [term 2 LEADER]: Received request to transfer leadership
W20240502 01:53:11.002241 26556 consensus_peers.cc:485] T e425e061e59a4bffa56b498fdf462a58 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 01:53:11.024780 30052 raft_consensus.cc:1050] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc: attempting to promote NON_VOTER aed397efb8844970b606bca15a77cb59 to VOTER
I20240502 01:53:11.025635 30052 consensus_queue.cc:237] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } }
I20240502 01:53:11.028625 27890 raft_consensus.cc:1261] T 895cfddfe1804d34a80b8a52e00efce2 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 2 FOLLOWER]: Refusing update from remote peer c292d733f9584891b46f52aafee403fc: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 01:53:11.029402 26622 raft_consensus.cc:1261] T 895cfddfe1804d34a80b8a52e00efce2 P d1e7d0eabafa46f382e19f596e2fc00d [term 2 FOLLOWER]: Refusing update from remote peer c292d733f9584891b46f52aafee403fc: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 01:53:11.029676 28025 raft_consensus.cc:1261] T 895cfddfe1804d34a80b8a52e00efce2 P aed397efb8844970b606bca15a77cb59 [term 2 LEARNER]: Refusing update from remote peer c292d733f9584891b46f52aafee403fc: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 01:53:11.030053 30052 consensus_queue.cc:1035] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:53:11.030452 30054 consensus_queue.cc:1035] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:53:11.030793 30051 consensus_queue.cc:1035] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:53:11.034096 30052 raft_consensus.cc:2932] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, aed397efb8844970b606bca15a77cb59 (127.25.109.135) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } }
I20240502 01:53:11.034664 26622 raft_consensus.cc:2932] T 895cfddfe1804d34a80b8a52e00efce2 P d1e7d0eabafa46f382e19f596e2fc00d [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, aed397efb8844970b606bca15a77cb59 (127.25.109.135) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } }
I20240502 01:53:11.036212 28025 raft_consensus.cc:2932] T 895cfddfe1804d34a80b8a52e00efce2 P aed397efb8844970b606bca15a77cb59 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, aed397efb8844970b606bca15a77cb59 (127.25.109.135) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } }
I20240502 01:53:11.039783 26078 catalog_manager.cc:5484] T 895cfddfe1804d34a80b8a52e00efce2 P d1e7d0eabafa46f382e19f596e2fc00d reported cstate change: config changed from index 11 to 12, aed397efb8844970b606bca15a77cb59 (127.25.109.135) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "c292d733f9584891b46f52aafee403fc" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } }
I20240502 01:53:11.042243 27890 raft_consensus.cc:2932] T 895cfddfe1804d34a80b8a52e00efce2 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, aed397efb8844970b606bca15a77cb59 (127.25.109.135) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } }
I20240502 01:53:11.045547 28160 consensus_queue.cc:237] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } }
I20240502 01:53:11.047480 28025 raft_consensus.cc:1261] T 895cfddfe1804d34a80b8a52e00efce2 P aed397efb8844970b606bca15a77cb59 [term 2 FOLLOWER]: Refusing update from remote peer c292d733f9584891b46f52aafee403fc: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 01:53:11.047536 27890 raft_consensus.cc:1261] T 895cfddfe1804d34a80b8a52e00efce2 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 2 FOLLOWER]: Refusing update from remote peer c292d733f9584891b46f52aafee403fc: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 01:53:11.048031 29965 consensus_queue.cc:1035] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:53:11.048396 30054 consensus_queue.cc:1035] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:53:11.050631 30051 raft_consensus.cc:2932] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } }
I20240502 01:53:11.051398 28025 raft_consensus.cc:2932] T 895cfddfe1804d34a80b8a52e00efce2 P aed397efb8844970b606bca15a77cb59 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } }
I20240502 01:53:11.051312 27890 raft_consensus.cc:2932] T 895cfddfe1804d34a80b8a52e00efce2 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } }
I20240502 01:53:11.055610 26066 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 895cfddfe1804d34a80b8a52e00efce2 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:11.056308 26078 catalog_manager.cc:5484] T 895cfddfe1804d34a80b8a52e00efce2 P 6ecf0a12e15f45a3bfa042f88c100a5e reported cstate change: config changed from index 12 to 13, VOTER d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132) evicted. New cstate: current_term: 2 leader_uuid: "c292d733f9584891b46f52aafee403fc" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: false } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } }
I20240502 01:53:11.060380 26601 tablet_service.cc:1514] Processing DeleteTablet for tablet 895cfddfe1804d34a80b8a52e00efce2 with delete_type TABLET_DATA_TOMBSTONED (TS d1e7d0eabafa46f382e19f596e2fc00d not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:59648
I20240502 01:53:11.061205 30058 tablet_replica.cc:330] T 895cfddfe1804d34a80b8a52e00efce2 P d1e7d0eabafa46f382e19f596e2fc00d: stopping tablet replica
I20240502 01:53:11.061447 30058 raft_consensus.cc:2229] T 895cfddfe1804d34a80b8a52e00efce2 P d1e7d0eabafa46f382e19f596e2fc00d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:11.061825 30058 raft_consensus.cc:2258] T 895cfddfe1804d34a80b8a52e00efce2 P d1e7d0eabafa46f382e19f596e2fc00d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:11.062674 30058 ts_tablet_manager.cc:1875] T 895cfddfe1804d34a80b8a52e00efce2 P d1e7d0eabafa46f382e19f596e2fc00d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:11.067471 30058 ts_tablet_manager.cc:1888] T 895cfddfe1804d34a80b8a52e00efce2 P d1e7d0eabafa46f382e19f596e2fc00d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.12
I20240502 01:53:11.067629 30058 log.cc:1196] T 895cfddfe1804d34a80b8a52e00efce2 P d1e7d0eabafa46f382e19f596e2fc00d: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal/wals/895cfddfe1804d34a80b8a52e00efce2
I20240502 01:53:11.068508 26067 catalog_manager.cc:4830] TS d1e7d0eabafa46f382e19f596e2fc00d (127.25.109.132:37695): tablet 895cfddfe1804d34a80b8a52e00efce2 (table rebalance_test_table_0 [id=45682f0e9b4f45c3b3e4cb8de40b23bd]) successfully deleted
W20240502 01:53:11.081684 26424 consensus_peers.cc:485] T ebe8589feed3400aa6bca73df420778d P 6c70fbdc8a0b42ad8c09c6dbc842cc96 -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:53:11.154975 26292 consensus_peers.cc:485] T 7def725255ba4448837b47ba65a6ed44 P 868e6e4358f4475bb45967b05eccb09f -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 01:53:11.210494 26356 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "38dbb02fca694854ab66393a87c84908"
dest_uuid: "868e6e4358f4475bb45967b05eccb09f"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:47588
I20240502 01:53:11.210750 26356 raft_consensus.cc:595] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [term 2 LEADER]: Received request to transfer leadership
I20240502 01:53:11.228462 26356 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "38dbb02fca694854ab66393a87c84908"
dest_uuid: "868e6e4358f4475bb45967b05eccb09f"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:47614
I20240502 01:53:11.228684 26356 raft_consensus.cc:595] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [term 2 LEADER]: Received request to transfer leadership
W20240502 01:53:11.266050 26424 consensus_peers.cc:485] T 64c74d4af3254bc28110d5b6d3b1c51f P 6c70fbdc8a0b42ad8c09c6dbc842cc96 -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:53:11.296463 26556 consensus_peers.cc:485] T 4e6854ffbc944e45b51256143505cdb8 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:53:11.300751 26292 consensus_peers.cc:485] T fbd456fceb0f4c6db45662eb619cfd21 P 868e6e4358f4475bb45967b05eccb09f -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 01:53:11.311924 30024 raft_consensus.cc:979] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f: : Instructing follower 6ecf0a12e15f45a3bfa042f88c100a5e to start an election
I20240502 01:53:11.312129 30024 raft_consensus.cc:1067] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [term 2 LEADER]: Signalling peer 6ecf0a12e15f45a3bfa042f88c100a5e to start an election
I20240502 01:53:11.312999 27890 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "38dbb02fca694854ab66393a87c84908"
dest_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e"
 from {username='slave'} at 127.25.109.130:60927
I20240502 01:53:11.313217 27890 raft_consensus.cc:482] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:53:11.313329 27890 raft_consensus.cc:3037] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:53:11.315301 27890 raft_consensus.cc:504] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: false } }
I20240502 01:53:11.318687 27890 leader_election.cc:288] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e [CANDIDATE]: Term 3 election: Requested vote from peers 868e6e4358f4475bb45967b05eccb09f (127.25.109.130:45571), aed397efb8844970b606bca15a77cb59 (127.25.109.135:39905), ca931e9360d44079b6059b8d9e7a6d20 (127.25.109.133:33081)
I20240502 01:53:11.319438 27756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbb02fca694854ab66393a87c84908" candidate_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: true dest_uuid: "ca931e9360d44079b6059b8d9e7a6d20"
I20240502 01:53:11.319767 27756 raft_consensus.cc:3037] T 38dbb02fca694854ab66393a87c84908 P ca931e9360d44079b6059b8d9e7a6d20 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:53:11.322965 27756 raft_consensus.cc:2454] T 38dbb02fca694854ab66393a87c84908 P ca931e9360d44079b6059b8d9e7a6d20 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ecf0a12e15f45a3bfa042f88c100a5e in term 3.
I20240502 01:53:11.329353 26356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbb02fca694854ab66393a87c84908" candidate_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: true dest_uuid: "868e6e4358f4475bb45967b05eccb09f"
I20240502 01:53:11.329617 26356 raft_consensus.cc:3032] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:53:11.329706 26356 raft_consensus.cc:729] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 868e6e4358f4475bb45967b05eccb09f, State: Running, Role: LEADER
I20240502 01:53:11.330041 26356 consensus_queue.cc:260] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: false } }
I20240502 01:53:11.330456 26356 raft_consensus.cc:3037] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:53:11.333086 26356 raft_consensus.cc:2454] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ecf0a12e15f45a3bfa042f88c100a5e in term 3.
I20240502 01:53:11.333675 27824 leader_election.cc:302] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6ecf0a12e15f45a3bfa042f88c100a5e, 868e6e4358f4475bb45967b05eccb09f, ca931e9360d44079b6059b8d9e7a6d20; no voters: 
I20240502 01:53:11.334003 29941 raft_consensus.cc:2781] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:53:11.334939 29941 raft_consensus.cc:686] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 3 LEADER]: Becoming Leader. State: Replica: 6ecf0a12e15f45a3bfa042f88c100a5e, State: Running, Role: LEADER
I20240502 01:53:11.335335 29941 consensus_queue.cc:237] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: false } }
I20240502 01:53:11.335675 28025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38dbb02fca694854ab66393a87c84908" candidate_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: true dest_uuid: "aed397efb8844970b606bca15a77cb59"
I20240502 01:53:11.336072 28025 raft_consensus.cc:3037] T 38dbb02fca694854ab66393a87c84908 P aed397efb8844970b606bca15a77cb59 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:53:11.338523 26080 catalog_manager.cc:5484] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e reported cstate change: term changed from 2 to 3, leader changed from 868e6e4358f4475bb45967b05eccb09f (127.25.109.130) to 6ecf0a12e15f45a3bfa042f88c100a5e (127.25.109.134). New cstate: current_term: 3 leader_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:11.338999 28025 raft_consensus.cc:2454] T 38dbb02fca694854ab66393a87c84908 P aed397efb8844970b606bca15a77cb59 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ecf0a12e15f45a3bfa042f88c100a5e in term 3.
W20240502 01:53:11.494539 26424 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 01:53:11.727259 28025 raft_consensus.cc:1261] T 38dbb02fca694854ab66393a87c84908 P aed397efb8844970b606bca15a77cb59 [term 3 FOLLOWER]: Refusing update from remote peer 6ecf0a12e15f45a3bfa042f88c100a5e: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240502 01:53:11.727945 29941 consensus_queue.cc:1035] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:53:11.781786 26356 raft_consensus.cc:1261] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [term 3 FOLLOWER]: Refusing update from remote peer 6ecf0a12e15f45a3bfa042f88c100a5e: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240502 01:53:11.782497 29941 consensus_queue.cc:1035] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:53:11.786772 27756 raft_consensus.cc:1261] T 38dbb02fca694854ab66393a87c84908 P ca931e9360d44079b6059b8d9e7a6d20 [term 3 FOLLOWER]: Refusing update from remote peer 6ecf0a12e15f45a3bfa042f88c100a5e: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240502 01:53:11.787497 29941 consensus_queue.cc:1035] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240502 01:53:11.793956 26556 consensus_peers.cc:485] T 01b7ac587d134a61b333489d937c0a21 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 01:53:12.323835 27890 consensus_queue.cc:237] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: false } }
I20240502 01:53:12.325659 28025 raft_consensus.cc:1261] T 38dbb02fca694854ab66393a87c84908 P aed397efb8844970b606bca15a77cb59 [term 3 FOLLOWER]: Refusing update from remote peer 6ecf0a12e15f45a3bfa042f88c100a5e: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240502 01:53:12.325835 27756 raft_consensus.cc:1261] T 38dbb02fca694854ab66393a87c84908 P ca931e9360d44079b6059b8d9e7a6d20 [term 3 FOLLOWER]: Refusing update from remote peer 6ecf0a12e15f45a3bfa042f88c100a5e: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240502 01:53:12.326261 30128 consensus_queue.cc:1035] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 01:53:12.326582 30127 consensus_queue.cc:1035] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 01:53:12.328771 30105 raft_consensus.cc:2932] T 38dbb02fca694854ab66393a87c84908 P 6ecf0a12e15f45a3bfa042f88c100a5e [term 3 LEADER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER 868e6e4358f4475bb45967b05eccb09f (127.25.109.130) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: false } } }
I20240502 01:53:12.329340 28025 raft_consensus.cc:2932] T 38dbb02fca694854ab66393a87c84908 P aed397efb8844970b606bca15a77cb59 [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER 868e6e4358f4475bb45967b05eccb09f (127.25.109.130) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: false } } }
I20240502 01:53:12.329680 27756 raft_consensus.cc:2932] T 38dbb02fca694854ab66393a87c84908 P ca931e9360d44079b6059b8d9e7a6d20 [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER 868e6e4358f4475bb45967b05eccb09f (127.25.109.130) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: false } } }
I20240502 01:53:12.333542 26066 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 38dbb02fca694854ab66393a87c84908 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:12.334657 26080 catalog_manager.cc:5484] T 38dbb02fca694854ab66393a87c84908 P aed397efb8844970b606bca15a77cb59 reported cstate change: config changed from index 12 to 14, VOTER 868e6e4358f4475bb45967b05eccb09f (127.25.109.130) evicted. New cstate: current_term: 3 leader_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "6ecf0a12e15f45a3bfa042f88c100a5e" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 43811 } attrs { promote: false } } peers { permanent_uuid: "aed397efb8844970b606bca15a77cb59" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 39905 } attrs { promote: false } } peers { permanent_uuid: "ca931e9360d44079b6059b8d9e7a6d20" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 33081 } attrs { promote: false } } }
I20240502 01:53:12.339347 26335 tablet_service.cc:1514] Processing DeleteTablet for tablet 38dbb02fca694854ab66393a87c84908 with delete_type TABLET_DATA_TOMBSTONED (TS 868e6e4358f4475bb45967b05eccb09f not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:47846
I20240502 01:53:12.340083 30134 tablet_replica.cc:330] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f: stopping tablet replica
I20240502 01:53:12.340517 30134 raft_consensus.cc:2229] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:12.340764 30134 raft_consensus.cc:2258] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:12.341898 30134 ts_tablet_manager.cc:1875] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:12.347143 30134 ts_tablet_manager.cc:1888] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.13
I20240502 01:53:12.347288 30134 log.cc:1196] T 38dbb02fca694854ab66393a87c84908 P 868e6e4358f4475bb45967b05eccb09f: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal/wals/38dbb02fca694854ab66393a87c84908
I20240502 01:53:12.348078 26067 catalog_manager.cc:4830] TS 868e6e4358f4475bb45967b05eccb09f (127.25.109.130:45571): tablet 38dbb02fca694854ab66393a87c84908 (table rebalance_test_table_3 [id=3dba071b1adb47bebabd07a94b201ce1]) successfully deleted
W20240502 01:53:12.363240 26292 consensus_peers.cc:485] T 062fa1dfab1046f287e2ddb1c240c4eb P 868e6e4358f4475bb45967b05eccb09f -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 01:53:12.612548 26335 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:53:12.626076 26467 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:12.667358 27869 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 01:53:12.668634 26292 consensus_peers.cc:485] T c32bdf0d644e499eaa2f755fab9abb77 P 868e6e4358f4475bb45967b05eccb09f -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 01:53:12.670368 27736 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:12.672806 28005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:53:12.689096 26556 consensus_peers.cc:485] T dc44aad1f4ab4469a29d280a11268be0 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 01:53:12.700786 28139 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:12.710359 26601 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:53:12.898303 26467 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:12.904614 26335 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:53:12.910766 27869 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:12.918691 28139 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:12.918965 27736 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:12.919196 28005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:53:12.948426 26556 consensus_peers.cc:485] T 536039b70452486684f7075cf45c1851 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 01:53:12.975101 26601 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:53:13.162375 26424 consensus_peers.cc:485] T ebe8589feed3400aa6bca73df420778d P 6c70fbdc8a0b42ad8c09c6dbc842cc96 -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 01:53:13.216990 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.129:44541
--local_ip_for_outbound_sockets=127.25.109.129
--tserver_master_addrs=127.25.109.190:44417
--webserver_port=41535
--webserver_interface=127.25.109.129
--builtin_ntp_servers=127.25.109.148:34611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240502 01:53:13.373602 26556 consensus_peers.cc:485] T e425e061e59a4bffa56b498fdf462a58 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:53:13.402671 26292 consensus_peers.cc:485] T 7def725255ba4448837b47ba65a6ed44 P 868e6e4358f4475bb45967b05eccb09f -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:53:13.586129 30213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:13.586463 30213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:13.586601 30213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:13.594313 30213 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:53:13.594476 30213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:13.594643 30213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.129
I20240502 01:53:13.604532 30213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:34611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.129:44541
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.25.109.129
--webserver_port=41535
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.25.109.190:44417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.129
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:13.605890 30213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:13.607928 30213 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:13.619352 26424 consensus_peers.cc:485] T 64c74d4af3254bc28110d5b6d3b1c51f P 6c70fbdc8a0b42ad8c09c6dbc842cc96 -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:53:13.621173 30219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:13.621692 30218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:13.622220 30221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:13.622799 30213 server_base.cc:1034] running on GCE node
I20240502 01:53:13.623425 30213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:13.624455 30213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:13.625667 30213 hybrid_clock.cc:648] HybridClock initialized: now 1714614793625612 us; error 44 us; skew 500 ppm
I20240502 01:53:13.626003 30213 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:13.628427 30213 webserver.cc:460] Webserver started at http://127.25.109.129:41535/ using document root <none> and password file <none>
I20240502 01:53:13.629215 30213 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:13.629349 30213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:13.635180 30213 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240502 01:53:13.639657 30227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:13.641274 30213 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.002s
I20240502 01:53:13.641484 30213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce"
format_stamp: "Formatted at 2024-05-02 01:52:27 on dist-test-slave-kj4b"
I20240502 01:53:13.642438 30213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:13.664749 30213 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:53:13.665663 30213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:13.665990 30213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:13.667026 30213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:13.669413 30234 ts_tablet_manager.cc:522] Loading tablet metadata (0/12 complete)
I20240502 01:53:13.697399 30213 ts_tablet_manager.cc:559] Loaded tablet metadata (12 total tablets, 12 live tablets)
I20240502 01:53:13.697543 30213 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.029s	user 0.002s	sys 0.000s
I20240502 01:53:13.697697 30213 ts_tablet_manager.cc:574] Registering tablets (0/12 complete)
I20240502 01:53:13.701521 30234 tablet_bootstrap.cc:492] T 01b7ac587d134a61b333489d937c0a21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap starting.
I20240502 01:53:13.721407 30213 ts_tablet_manager.cc:590] Registered 12 tablets
I20240502 01:53:13.721532 30213 ts_tablet_manager.cc:569] Time spent register tablets: real 0.024s	user 0.019s	sys 0.003s
I20240502 01:53:13.722487 30234 log.cc:824] T 01b7ac587d134a61b333489d937c0a21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:13.783552 30234 tablet_bootstrap.cc:492] T 01b7ac587d134a61b333489d937c0a21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:13.784643 30234 tablet_bootstrap.cc:492] T 01b7ac587d134a61b333489d937c0a21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap complete.
I20240502 01:53:13.786497 30234 ts_tablet_manager.cc:1367] T 01b7ac587d134a61b333489d937c0a21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent bootstrapping tablet: real 0.085s	user 0.070s	sys 0.008s
W20240502 01:53:13.797077 26556 consensus_peers.cc:485] T 4e6854ffbc944e45b51256143505cdb8 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 01:53:13.797155 30234 raft_consensus.cc:348] T 01b7ac587d134a61b333489d937c0a21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:53:13.798656 30234 raft_consensus.cc:729] T 01b7ac587d134a61b333489d937c0a21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f215e8fdc8cb45f9a5eff7cb9e6ccfce, State: Initialized, Role: FOLLOWER
I20240502 01:53:13.799719 30234 consensus_queue.cc:260] T 01b7ac587d134a61b333489d937c0a21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:53:13.804677 30234 ts_tablet_manager.cc:1398] T 01b7ac587d134a61b333489d937c0a21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent starting tablet: real 0.018s	user 0.018s	sys 0.000s
I20240502 01:53:13.805310 30234 tablet_bootstrap.cc:492] T ebe8589feed3400aa6bca73df420778d P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap starting.
I20240502 01:53:13.815131 30213 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.129:44541
I20240502 01:53:13.815151 30343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.129:44541 every 8 connection(s)
I20240502 01:53:13.816956 30213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 01:53:13.823385 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 30213
I20240502 01:53:13.863274 30345 heartbeater.cc:347] Connected to a master server at 127.25.109.190:44417
I20240502 01:53:13.863641 30345 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:13.864900 30345 heartbeater.cc:511] Master 127.25.109.190:44417 requested a full tablet report, sending...
I20240502 01:53:13.868809 26078 ts_manager.cc:194] Re-registered known tserver with Master: f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541)
I20240502 01:53:13.876286 26078 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.129:48615
I20240502 01:53:13.879947 30345 heartbeater.cc:503] Master 127.25.109.190:44417 was elected leader, sending a full tablet report...
I20240502 01:53:13.885706 30234 tablet_bootstrap.cc:492] T ebe8589feed3400aa6bca73df420778d P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:13.886278 30234 tablet_bootstrap.cc:492] T ebe8589feed3400aa6bca73df420778d P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap complete.
I20240502 01:53:13.887435 30234 ts_tablet_manager.cc:1367] T ebe8589feed3400aa6bca73df420778d P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent bootstrapping tablet: real 0.082s	user 0.046s	sys 0.016s
I20240502 01:53:13.888566 30234 raft_consensus.cc:348] T ebe8589feed3400aa6bca73df420778d P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } }
I20240502 01:53:13.888790 30234 raft_consensus.cc:729] T ebe8589feed3400aa6bca73df420778d P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f215e8fdc8cb45f9a5eff7cb9e6ccfce, State: Initialized, Role: FOLLOWER
I20240502 01:53:13.889058 30234 consensus_queue.cc:260] T ebe8589feed3400aa6bca73df420778d P f215e8fdc8cb45f9a5eff7cb9e6ccfce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } }
I20240502 01:53:13.889516 30234 ts_tablet_manager.cc:1398] T ebe8589feed3400aa6bca73df420778d P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:53:13.889775 30234 tablet_bootstrap.cc:492] T e425e061e59a4bffa56b498fdf462a58 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap starting.
I20240502 01:53:13.928516 30234 tablet_bootstrap.cc:492] T e425e061e59a4bffa56b498fdf462a58 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:13.928984 30234 tablet_bootstrap.cc:492] T e425e061e59a4bffa56b498fdf462a58 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap complete.
I20240502 01:53:13.929847 30234 ts_tablet_manager.cc:1367] T e425e061e59a4bffa56b498fdf462a58 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent bootstrapping tablet: real 0.040s	user 0.030s	sys 0.007s
I20240502 01:53:13.930532 30234 raft_consensus.cc:348] T e425e061e59a4bffa56b498fdf462a58 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:53:13.930727 30234 raft_consensus.cc:729] T e425e061e59a4bffa56b498fdf462a58 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f215e8fdc8cb45f9a5eff7cb9e6ccfce, State: Initialized, Role: FOLLOWER
I20240502 01:53:13.930971 30234 consensus_queue.cc:260] T e425e061e59a4bffa56b498fdf462a58 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:53:13.931489 30234 ts_tablet_manager.cc:1398] T e425e061e59a4bffa56b498fdf462a58 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:53:13.931905 30234 tablet_bootstrap.cc:492] T 7aa623407e7844989e7bf34db5d6ba35 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap starting.
I20240502 01:53:13.937822 30298 raft_consensus.cc:3037] T e425e061e59a4bffa56b498fdf462a58 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:53:13.973117 26292 consensus_peers.cc:485] T fbd456fceb0f4c6db45662eb619cfd21 P 868e6e4358f4475bb45967b05eccb09f -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 46: this message will repeat every 5th retry.
I20240502 01:53:13.978149 30234 tablet_bootstrap.cc:492] T 7aa623407e7844989e7bf34db5d6ba35 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:13.978667 30234 tablet_bootstrap.cc:492] T 7aa623407e7844989e7bf34db5d6ba35 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap complete.
I20240502 01:53:13.979548 30234 ts_tablet_manager.cc:1367] T 7aa623407e7844989e7bf34db5d6ba35 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent bootstrapping tablet: real 0.048s	user 0.024s	sys 0.019s
I20240502 01:53:13.980325 30234 raft_consensus.cc:348] T 7aa623407e7844989e7bf34db5d6ba35 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: NON_VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: true } }
I20240502 01:53:13.980592 30234 raft_consensus.cc:729] T 7aa623407e7844989e7bf34db5d6ba35 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f215e8fdc8cb45f9a5eff7cb9e6ccfce, State: Initialized, Role: FOLLOWER
I20240502 01:53:13.980865 30234 consensus_queue.cc:260] T 7aa623407e7844989e7bf34db5d6ba35 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } attrs { replace: true } } peers { permanent_uuid: "c292d733f9584891b46f52aafee403fc" member_type: NON_VOTER last_known_addr { host: "127.25.109.136" port: 38319 } attrs { promote: true } }
I20240502 01:53:13.981413 30234 ts_tablet_manager.cc:1398] T 7aa623407e7844989e7bf34db5d6ba35 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent starting tablet: real 0.002s	user 0.005s	sys 0.000s
I20240502 01:53:13.981730 30234 tablet_bootstrap.cc:492] T c32bdf0d644e499eaa2f755fab9abb77 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap starting.
I20240502 01:53:14.023475 30234 tablet_bootstrap.cc:492] T c32bdf0d644e499eaa2f755fab9abb77 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:14.023979 30234 tablet_bootstrap.cc:492] T c32bdf0d644e499eaa2f755fab9abb77 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap complete.
I20240502 01:53:14.024849 30234 ts_tablet_manager.cc:1367] T c32bdf0d644e499eaa2f755fab9abb77 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent bootstrapping tablet: real 0.043s	user 0.030s	sys 0.012s
I20240502 01:53:14.025550 30234 raft_consensus.cc:348] T c32bdf0d644e499eaa2f755fab9abb77 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } }
I20240502 01:53:14.025761 30234 raft_consensus.cc:729] T c32bdf0d644e499eaa2f755fab9abb77 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f215e8fdc8cb45f9a5eff7cb9e6ccfce, State: Initialized, Role: FOLLOWER
I20240502 01:53:14.026000 30234 consensus_queue.cc:260] T c32bdf0d644e499eaa2f755fab9abb77 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } }
I20240502 01:53:14.026517 30234 ts_tablet_manager.cc:1398] T c32bdf0d644e499eaa2f755fab9abb77 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:53:14.026788 30234 tablet_bootstrap.cc:492] T 536039b70452486684f7075cf45c1851 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap starting.
I20240502 01:53:14.082386 30234 tablet_bootstrap.cc:492] T 536039b70452486684f7075cf45c1851 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:14.083017 30234 tablet_bootstrap.cc:492] T 536039b70452486684f7075cf45c1851 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap complete.
I20240502 01:53:14.084290 30234 ts_tablet_manager.cc:1367] T 536039b70452486684f7075cf45c1851 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent bootstrapping tablet: real 0.058s	user 0.048s	sys 0.007s
I20240502 01:53:14.085196 30234 raft_consensus.cc:348] T 536039b70452486684f7075cf45c1851 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } }
I20240502 01:53:14.085472 30234 raft_consensus.cc:729] T 536039b70452486684f7075cf45c1851 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f215e8fdc8cb45f9a5eff7cb9e6ccfce, State: Initialized, Role: FOLLOWER
I20240502 01:53:14.085762 30234 consensus_queue.cc:260] T 536039b70452486684f7075cf45c1851 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } }
I20240502 01:53:14.086341 30234 ts_tablet_manager.cc:1398] T 536039b70452486684f7075cf45c1851 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:53:14.086656 30234 tablet_bootstrap.cc:492] T dc44aad1f4ab4469a29d280a11268be0 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap starting.
I20240502 01:53:14.144593 30234 tablet_bootstrap.cc:492] T dc44aad1f4ab4469a29d280a11268be0 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:14.145188 30234 tablet_bootstrap.cc:492] T dc44aad1f4ab4469a29d280a11268be0 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap complete.
I20240502 01:53:14.146438 30234 ts_tablet_manager.cc:1367] T dc44aad1f4ab4469a29d280a11268be0 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent bootstrapping tablet: real 0.060s	user 0.041s	sys 0.015s
I20240502 01:53:14.147444 30234 raft_consensus.cc:348] T dc44aad1f4ab4469a29d280a11268be0 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:53:14.147919 30234 raft_consensus.cc:729] T dc44aad1f4ab4469a29d280a11268be0 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f215e8fdc8cb45f9a5eff7cb9e6ccfce, State: Initialized, Role: FOLLOWER
I20240502 01:53:14.148223 30234 consensus_queue.cc:260] T dc44aad1f4ab4469a29d280a11268be0 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:53:14.148969 30234 ts_tablet_manager.cc:1398] T dc44aad1f4ab4469a29d280a11268be0 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.004s
I20240502 01:53:14.149317 30234 tablet_bootstrap.cc:492] T 7def725255ba4448837b47ba65a6ed44 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap starting.
I20240502 01:53:14.189219 30234 tablet_bootstrap.cc:492] T 7def725255ba4448837b47ba65a6ed44 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:14.189852 30234 tablet_bootstrap.cc:492] T 7def725255ba4448837b47ba65a6ed44 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap complete.
I20240502 01:53:14.191030 30234 ts_tablet_manager.cc:1367] T 7def725255ba4448837b47ba65a6ed44 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent bootstrapping tablet: real 0.042s	user 0.018s	sys 0.020s
I20240502 01:53:14.191785 30234 raft_consensus.cc:348] T 7def725255ba4448837b47ba65a6ed44 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } }
I20240502 01:53:14.192023 30234 raft_consensus.cc:729] T 7def725255ba4448837b47ba65a6ed44 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f215e8fdc8cb45f9a5eff7cb9e6ccfce, State: Initialized, Role: FOLLOWER
I20240502 01:53:14.192294 30234 consensus_queue.cc:260] T 7def725255ba4448837b47ba65a6ed44 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } }
I20240502 01:53:14.192848 30234 ts_tablet_manager.cc:1398] T 7def725255ba4448837b47ba65a6ed44 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240502 01:53:14.193200 30234 tablet_bootstrap.cc:492] T 062fa1dfab1046f287e2ddb1c240c4eb P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap starting.
I20240502 01:53:14.236263 30234 tablet_bootstrap.cc:492] T 062fa1dfab1046f287e2ddb1c240c4eb P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:14.236738 30234 tablet_bootstrap.cc:492] T 062fa1dfab1046f287e2ddb1c240c4eb P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap complete.
I20240502 01:53:14.237565 30234 ts_tablet_manager.cc:1367] T 062fa1dfab1046f287e2ddb1c240c4eb P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent bootstrapping tablet: real 0.044s	user 0.031s	sys 0.012s
I20240502 01:53:14.238268 30234 raft_consensus.cc:348] T 062fa1dfab1046f287e2ddb1c240c4eb P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:53:14.238464 30234 raft_consensus.cc:729] T 062fa1dfab1046f287e2ddb1c240c4eb P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f215e8fdc8cb45f9a5eff7cb9e6ccfce, State: Initialized, Role: FOLLOWER
I20240502 01:53:14.238739 30234 consensus_queue.cc:260] T 062fa1dfab1046f287e2ddb1c240c4eb P f215e8fdc8cb45f9a5eff7cb9e6ccfce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:53:14.239282 30234 ts_tablet_manager.cc:1398] T 062fa1dfab1046f287e2ddb1c240c4eb P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
I20240502 01:53:14.239630 30234 tablet_bootstrap.cc:492] T 4e6854ffbc944e45b51256143505cdb8 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap starting.
I20240502 01:53:14.283419 30234 tablet_bootstrap.cc:492] T 4e6854ffbc944e45b51256143505cdb8 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:14.283941 30234 tablet_bootstrap.cc:492] T 4e6854ffbc944e45b51256143505cdb8 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap complete.
I20240502 01:53:14.284801 30234 ts_tablet_manager.cc:1367] T 4e6854ffbc944e45b51256143505cdb8 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent bootstrapping tablet: real 0.045s	user 0.037s	sys 0.007s
I20240502 01:53:14.285487 30234 raft_consensus.cc:348] T 4e6854ffbc944e45b51256143505cdb8 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:53:14.285665 30234 raft_consensus.cc:729] T 4e6854ffbc944e45b51256143505cdb8 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f215e8fdc8cb45f9a5eff7cb9e6ccfce, State: Initialized, Role: FOLLOWER
I20240502 01:53:14.285916 30234 consensus_queue.cc:260] T 4e6854ffbc944e45b51256143505cdb8 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "d1e7d0eabafa46f382e19f596e2fc00d" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 37695 } }
I20240502 01:53:14.286423 30234 ts_tablet_manager.cc:1398] T 4e6854ffbc944e45b51256143505cdb8 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:53:14.286717 30234 tablet_bootstrap.cc:492] T 64c74d4af3254bc28110d5b6d3b1c51f P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap starting.
I20240502 01:53:14.306354 30298 raft_consensus.cc:3037] T ebe8589feed3400aa6bca73df420778d P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:14.343947 30234 tablet_bootstrap.cc:492] T 64c74d4af3254bc28110d5b6d3b1c51f P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:14.344508 30234 tablet_bootstrap.cc:492] T 64c74d4af3254bc28110d5b6d3b1c51f P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap complete.
I20240502 01:53:14.346642 30234 ts_tablet_manager.cc:1367] T 64c74d4af3254bc28110d5b6d3b1c51f P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent bootstrapping tablet: real 0.060s	user 0.046s	sys 0.012s
I20240502 01:53:14.347625 30234 raft_consensus.cc:348] T 64c74d4af3254bc28110d5b6d3b1c51f P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:53:14.347909 30234 raft_consensus.cc:729] T 64c74d4af3254bc28110d5b6d3b1c51f P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f215e8fdc8cb45f9a5eff7cb9e6ccfce, State: Initialized, Role: FOLLOWER
I20240502 01:53:14.348178 30234 consensus_queue.cc:260] T 64c74d4af3254bc28110d5b6d3b1c51f P f215e8fdc8cb45f9a5eff7cb9e6ccfce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:53:14.350915 30234 ts_tablet_manager.cc:1398] T 64c74d4af3254bc28110d5b6d3b1c51f P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.002s
I20240502 01:53:14.351215 30234 tablet_bootstrap.cc:492] T fbd456fceb0f4c6db45662eb619cfd21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap starting.
I20240502 01:53:14.402575 30234 tablet_bootstrap.cc:492] T fbd456fceb0f4c6db45662eb619cfd21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:14.403182 30234 tablet_bootstrap.cc:492] T fbd456fceb0f4c6db45662eb619cfd21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Bootstrap complete.
I20240502 01:53:14.404500 30234 ts_tablet_manager.cc:1367] T fbd456fceb0f4c6db45662eb619cfd21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent bootstrapping tablet: real 0.053s	user 0.032s	sys 0.020s
I20240502 01:53:14.405285 30234 raft_consensus.cc:348] T fbd456fceb0f4c6db45662eb619cfd21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:53:14.405570 30234 raft_consensus.cc:729] T fbd456fceb0f4c6db45662eb619cfd21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f215e8fdc8cb45f9a5eff7cb9e6ccfce, State: Initialized, Role: FOLLOWER
I20240502 01:53:14.405808 30234 consensus_queue.cc:260] T fbd456fceb0f4c6db45662eb619cfd21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c70fbdc8a0b42ad8c09c6dbc842cc96" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 38693 } } peers { permanent_uuid: "f215e8fdc8cb45f9a5eff7cb9e6ccfce" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 44541 } } peers { permanent_uuid: "868e6e4358f4475bb45967b05eccb09f" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 45571 } }
I20240502 01:53:14.406392 30234 ts_tablet_manager.cc:1398] T fbd456fceb0f4c6db45662eb619cfd21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
I20240502 01:53:14.409332 30297 raft_consensus.cc:3037] T fbd456fceb0f4c6db45662eb619cfd21 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:14.469451 27869 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:14.481984 26335 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:53:14.481974 28005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:14.484392 28139 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:14.500371 26601 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:53:14.501284 26467 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:14.504626 27736 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:14.516607 30278 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:14.708374 30297 raft_consensus.cc:3037] T 64c74d4af3254bc28110d5b6d3b1c51f P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:14.726255 26467 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:14.732069 27869 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:14.742946 26335 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:53:14.755223 28005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:14.764719 28139 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:14.792670 27736 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:14.809871 30297 raft_consensus.cc:3037] T 4e6854ffbc944e45b51256143505cdb8 P f215e8fdc8cb45f9a5eff7cb9e6ccfce [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:14.824369 26601 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:53:14.856639 30278 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:15.033658 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 30213
I20240502 01:53:15.040632 30339 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:15.154425 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 30213
I20240502 01:53:15.174654 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 26274
I20240502 01:53:15.180420 26397 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 01:53:15.303656 26556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111) [suppressed 36 similar messages]
W20240502 01:53:15.305302 26556 consensus_peers.cc:485] T 01b7ac587d134a61b333489d937c0a21 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:53:15.340359 26556 consensus_peers.cc:485] T 4e6854ffbc944e45b51256143505cdb8 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:15.364022 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 26274
W20240502 01:53:15.388032 26556 consensus_peers.cc:485] T 536039b70452486684f7075cf45c1851 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:53:15.389644 26555 connection.cc:535] server connection from 127.25.109.130:59951 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:53:15.390367 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 26405
W20240502 01:53:15.394482 26556 consensus_peers.cc:485] T dc44aad1f4ab4469a29d280a11268be0 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer f215e8fdc8cb45f9a5eff7cb9e6ccfce (127.25.109.129:44541): Couldn't send request to peer f215e8fdc8cb45f9a5eff7cb9e6ccfce. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:44541: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:15.396651 26530 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 01:53:15.435195 26555 consensus_peers.cc:485] T 01b7ac587d134a61b333489d937c0a21 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer 868e6e4358f4475bb45967b05eccb09f (127.25.109.130:45571): Couldn't send request to peer 868e6e4358f4475bb45967b05eccb09f. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.130:45571: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:15.576889 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 26405
W20240502 01:53:15.601540 26555 consensus_peers.cc:485] T dc44aad1f4ab4469a29d280a11268be0 P d1e7d0eabafa46f382e19f596e2fc00d -> Peer 868e6e4358f4475bb45967b05eccb09f (127.25.109.130:45571): Couldn't send request to peer 868e6e4358f4475bb45967b05eccb09f. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.130:45571: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:15.606235 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 26538
I20240502 01:53:15.612262 26663 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:15.793552 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 26538
I20240502 01:53:15.819962 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 27671
I20240502 01:53:15.826048 27796 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:15.954962 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 27671
I20240502 01:53:15.974785 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 27807
I20240502 01:53:15.980916 27932 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:16.122531 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 27807
W20240502 01:53:16.143142 28091 connection.cc:535] client connection to 127.25.109.134:43811 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:53:16.143463 28091 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:53:16.143599 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 27940
W20240502 01:53:16.145371 28091 consensus_peers.cc:485] T 895cfddfe1804d34a80b8a52e00efce2 P c292d733f9584891b46f52aafee403fc -> Peer 6ecf0a12e15f45a3bfa042f88c100a5e (127.25.109.134:43811): Couldn't send request to peer 6ecf0a12e15f45a3bfa042f88c100a5e. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:43811: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:16.150171 28066 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:16.279954 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 27940
I20240502 01:53:16.300268 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 28075
I20240502 01:53:16.306921 28201 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:16.439312 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 28075
I20240502 01:53:16.460476 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 26050
I20240502 01:53:16.461633 26110 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:16.633541 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 26050
2024-05-02T01:53:16Z chronyd exiting
[       OK ] IgnoredTserverGoesDownDuringRebalancingTest.TserverDown (50347 ms)
[----------] 1 test from IgnoredTserverGoesDownDuringRebalancingTest (50347 ms total)

[----------] 1 test from RebalanceStartCriteriaTest
[ RUN      ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/0
I20240502 01:53:16.697468 26038 test_util.cc:268] Using random seed: -572375213
I20240502 01:53:16.699342 26038 ts_itest-base.cc:112] Starting cluster with:
I20240502 01:53:16.699460 26038 ts_itest-base.cc:113] --------------
I20240502 01:53:16.699517 26038 ts_itest-base.cc:114] 5 tablet servers
I20240502 01:53:16.699582 26038 ts_itest-base.cc:115] 3 replicas per TS
I20240502 01:53:16.699631 26038 ts_itest-base.cc:116] --------------
2024-05-02T01:53:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T01:53:16Z Disabled control of system clock
I20240502 01:53:16.723037 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.190:35257
--webserver_interface=127.25.109.190
--webserver_port=0
--builtin_ntp_servers=127.25.109.148:38401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.190:35257
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240502 01:53:17.051436 30494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:17.051764 30494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:17.051929 30494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:17.059424 30494 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:53:17.059573 30494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 01:53:17.059646 30494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:17.059729 30494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 01:53:17.059793 30494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 01:53:17.069099 30494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:38401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.25.109.190:35257
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.190:35257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.25.109.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:17.070469 30494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:17.072278 30494 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:17.083000 30500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:17.083581 30499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:17.084012 30502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:17.084594 30494 server_base.cc:1034] running on GCE node
I20240502 01:53:17.085409 30494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:17.086901 30494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:17.088109 30494 hybrid_clock.cc:648] HybridClock initialized: now 1714614797088061 us; error 35 us; skew 500 ppm
I20240502 01:53:17.088476 30494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:17.090785 30494 webserver.cc:460] Webserver started at http://127.25.109.190:44571/ using document root <none> and password file <none>
I20240502 01:53:17.091545 30494 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:17.091651 30494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:17.092041 30494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:17.094719 30494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "de093ffd1df348f48a2723449312a1e6"
format_stamp: "Formatted at 2024-05-02 01:53:17 on dist-test-slave-kj4b"
I20240502 01:53:17.095616 30494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "de093ffd1df348f48a2723449312a1e6"
format_stamp: "Formatted at 2024-05-02 01:53:17 on dist-test-slave-kj4b"
I20240502 01:53:17.101132 30494 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.000s
I20240502 01:53:17.104727 30508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:17.106149 30494 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240502 01:53:17.106343 30494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal
uuid: "de093ffd1df348f48a2723449312a1e6"
format_stamp: "Formatted at 2024-05-02 01:53:17 on dist-test-slave-kj4b"
I20240502 01:53:17.106549 30494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:17.146430 30494 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:53:17.147337 30494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:17.147634 30494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:17.165416 30494 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.190:35257
I20240502 01:53:17.165450 30561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.190:35257 every 8 connection(s)
I20240502 01:53:17.166842 30494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 01:53:17.169867 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 30494
I20240502 01:53:17.170115 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 01:53:17.171293 30562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:17.184289 30562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de093ffd1df348f48a2723449312a1e6: Bootstrap starting.
I20240502 01:53:17.187350 30562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de093ffd1df348f48a2723449312a1e6: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:17.188411 30562 log.cc:824] T 00000000000000000000000000000000 P de093ffd1df348f48a2723449312a1e6: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:17.190966 30562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de093ffd1df348f48a2723449312a1e6: No bootstrap required, opened a new log
I20240502 01:53:17.197201 30562 raft_consensus.cc:348] T 00000000000000000000000000000000 P de093ffd1df348f48a2723449312a1e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de093ffd1df348f48a2723449312a1e6" member_type: VOTER last_known_addr { host: "127.25.109.190" port: 35257 } }
I20240502 01:53:17.197568 30562 raft_consensus.cc:374] T 00000000000000000000000000000000 P de093ffd1df348f48a2723449312a1e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:17.197678 30562 raft_consensus.cc:729] T 00000000000000000000000000000000 P de093ffd1df348f48a2723449312a1e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de093ffd1df348f48a2723449312a1e6, State: Initialized, Role: FOLLOWER
I20240502 01:53:17.198385 30562 consensus_queue.cc:260] T 00000000000000000000000000000000 P de093ffd1df348f48a2723449312a1e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de093ffd1df348f48a2723449312a1e6" member_type: VOTER last_known_addr { host: "127.25.109.190" port: 35257 } }
I20240502 01:53:17.198619 30562 raft_consensus.cc:388] T 00000000000000000000000000000000 P de093ffd1df348f48a2723449312a1e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:53:17.198763 30562 raft_consensus.cc:482] T 00000000000000000000000000000000 P de093ffd1df348f48a2723449312a1e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:53:17.198931 30562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de093ffd1df348f48a2723449312a1e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:17.201130 30562 raft_consensus.cc:504] T 00000000000000000000000000000000 P de093ffd1df348f48a2723449312a1e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de093ffd1df348f48a2723449312a1e6" member_type: VOTER last_known_addr { host: "127.25.109.190" port: 35257 } }
I20240502 01:53:17.201660 30562 leader_election.cc:302] T 00000000000000000000000000000000 P de093ffd1df348f48a2723449312a1e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de093ffd1df348f48a2723449312a1e6; no voters: 
I20240502 01:53:17.202338 30562 leader_election.cc:288] T 00000000000000000000000000000000 P de093ffd1df348f48a2723449312a1e6 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 01:53:17.202482 30567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de093ffd1df348f48a2723449312a1e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:17.203056 30567 raft_consensus.cc:686] T 00000000000000000000000000000000 P de093ffd1df348f48a2723449312a1e6 [term 1 LEADER]: Becoming Leader. State: Replica: de093ffd1df348f48a2723449312a1e6, State: Running, Role: LEADER
I20240502 01:53:17.203697 30567 consensus_queue.cc:237] T 00000000000000000000000000000000 P de093ffd1df348f48a2723449312a1e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de093ffd1df348f48a2723449312a1e6" member_type: VOTER last_known_addr { host: "127.25.109.190" port: 35257 } }
I20240502 01:53:17.204668 30562 sys_catalog.cc:564] T 00000000000000000000000000000000 P de093ffd1df348f48a2723449312a1e6 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:53:17.206720 30568 sys_catalog.cc:455] T 00000000000000000000000000000000 P de093ffd1df348f48a2723449312a1e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de093ffd1df348f48a2723449312a1e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de093ffd1df348f48a2723449312a1e6" member_type: VOTER last_known_addr { host: "127.25.109.190" port: 35257 } } }
I20240502 01:53:17.207080 30568 sys_catalog.cc:458] T 00000000000000000000000000000000 P de093ffd1df348f48a2723449312a1e6 [sys.catalog]: This master's current role is: LEADER
I20240502 01:53:17.207463 30569 sys_catalog.cc:455] T 00000000000000000000000000000000 P de093ffd1df348f48a2723449312a1e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de093ffd1df348f48a2723449312a1e6. Latest consensus state: current_term: 1 leader_uuid: "de093ffd1df348f48a2723449312a1e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de093ffd1df348f48a2723449312a1e6" member_type: VOTER last_known_addr { host: "127.25.109.190" port: 35257 } } }
I20240502 01:53:17.207659 30569 sys_catalog.cc:458] T 00000000000000000000000000000000 P de093ffd1df348f48a2723449312a1e6 [sys.catalog]: This master's current role is: LEADER
I20240502 01:53:17.209224 30575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:53:17.214959 30575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:53:17.226078 30575 catalog_manager.cc:1347] Generated new cluster ID: a0e8c2e490a3428093908b2c300d8398
I20240502 01:53:17.226228 30575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:53:17.234338 30575 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:53:17.235653 30575 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:53:17.243182 30575 catalog_manager.cc:5857] T 00000000000000000000000000000000 P de093ffd1df348f48a2723449312a1e6: Generated new TSK 0
I20240502 01:53:17.244235 30575 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:53:17.252463 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.129:0
--local_ip_for_outbound_sockets=127.25.109.129
--webserver_interface=127.25.109.129
--webserver_port=0
--tserver_master_addrs=127.25.109.190:35257
--builtin_ntp_servers=127.25.109.148:38401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 01:53:17.590354 30586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:17.590672 30586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:17.590765 30586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:17.598423 30586 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:53:17.598560 30586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:17.598675 30586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.129
I20240502 01:53:17.608058 30586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:38401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.25.109.129
--webserver_port=0
--tserver_master_addrs=127.25.109.190:35257
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.129
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:17.609783 30586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:17.611596 30586 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:17.624004 30591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:17.624455 30594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:17.624936 30592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:17.625499 30586 server_base.cc:1034] running on GCE node
I20240502 01:53:17.626085 30586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:17.627043 30586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:17.628244 30586 hybrid_clock.cc:648] HybridClock initialized: now 1714614797628186 us; error 56 us; skew 500 ppm
I20240502 01:53:17.628561 30586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:17.636545 30586 webserver.cc:460] Webserver started at http://127.25.109.129:38101/ using document root <none> and password file <none>
I20240502 01:53:17.637344 30586 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:17.637475 30586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:17.637805 30586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:17.640537 30586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "d96d46cf42454fb9ba48f87dc8d8dd2f"
format_stamp: "Formatted at 2024-05-02 01:53:17 on dist-test-slave-kj4b"
I20240502 01:53:17.641311 30586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "d96d46cf42454fb9ba48f87dc8d8dd2f"
format_stamp: "Formatted at 2024-05-02 01:53:17 on dist-test-slave-kj4b"
I20240502 01:53:17.646620 30586 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.002s	sys 0.003s
I20240502 01:53:17.650197 30600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:17.651605 30586 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:53:17.651773 30586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "d96d46cf42454fb9ba48f87dc8d8dd2f"
format_stamp: "Formatted at 2024-05-02 01:53:17 on dist-test-slave-kj4b"
I20240502 01:53:17.652016 30586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:17.686846 30586 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:53:17.687789 30586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:17.688102 30586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:17.689064 30586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:17.690698 30586 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:17.690795 30586 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:17.690912 30586 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:17.690981 30586 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:17.721853 30586 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.129:35971
I20240502 01:53:17.721930 30714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.129:35971 every 8 connection(s)
I20240502 01:53:17.723222 30586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 01:53:17.733702 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 30586
I20240502 01:53:17.733933 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 01:53:17.736514 30715 heartbeater.cc:347] Connected to a master server at 127.25.109.190:35257
I20240502 01:53:17.736841 30715 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:17.737607 30715 heartbeater.cc:511] Master 127.25.109.190:35257 requested a full tablet report, sending...
I20240502 01:53:17.737762 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.130:0
--local_ip_for_outbound_sockets=127.25.109.130
--webserver_interface=127.25.109.130
--webserver_port=0
--tserver_master_addrs=127.25.109.190:35257
--builtin_ntp_servers=127.25.109.148:38401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:53:17.739703 30524 ts_manager.cc:194] Registered new tserver with Master: d96d46cf42454fb9ba48f87dc8d8dd2f (127.25.109.129:35971)
I20240502 01:53:17.741338 30524 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.129:50915
W20240502 01:53:18.059509 30720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:18.059816 30720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:18.059965 30720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:18.067524 30720 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:53:18.067664 30720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:18.067838 30720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.130
I20240502 01:53:18.077075 30720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:38401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.25.109.130
--webserver_port=0
--tserver_master_addrs=127.25.109.190:35257
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.130
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:18.078397 30720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:18.080188 30720 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:18.092681 30725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:18.093274 30726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:18.093647 30728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:18.094287 30720 server_base.cc:1034] running on GCE node
I20240502 01:53:18.094944 30720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:18.095983 30720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:18.097194 30720 hybrid_clock.cc:648] HybridClock initialized: now 1714614798097137 us; error 56 us; skew 500 ppm
I20240502 01:53:18.097524 30720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:18.099927 30720 webserver.cc:460] Webserver started at http://127.25.109.130:34177/ using document root <none> and password file <none>
I20240502 01:53:18.100734 30720 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:18.100855 30720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:18.101187 30720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:18.103993 30720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "ab67584cb8b5451f9575a56d3593f032"
format_stamp: "Formatted at 2024-05-02 01:53:18 on dist-test-slave-kj4b"
I20240502 01:53:18.104838 30720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "ab67584cb8b5451f9575a56d3593f032"
format_stamp: "Formatted at 2024-05-02 01:53:18 on dist-test-slave-kj4b"
I20240502 01:53:18.110440 30720 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.001s	sys 0.004s
I20240502 01:53:18.114040 30734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:18.115499 30720 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:53:18.115667 30720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ab67584cb8b5451f9575a56d3593f032"
format_stamp: "Formatted at 2024-05-02 01:53:18 on dist-test-slave-kj4b"
I20240502 01:53:18.115897 30720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:18.136628 30720 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:53:18.137534 30720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:18.137814 30720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:18.138831 30720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:18.140488 30720 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:18.140589 30720 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:18.140702 30720 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:18.140774 30720 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:18.171257 30720 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.130:43749
I20240502 01:53:18.171339 30849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.130:43749 every 8 connection(s)
I20240502 01:53:18.172602 30720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 01:53:18.175719 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 30720
I20240502 01:53:18.175973 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 01:53:18.185075 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.131:0
--local_ip_for_outbound_sockets=127.25.109.131
--webserver_interface=127.25.109.131
--webserver_port=0
--tserver_master_addrs=127.25.109.190:35257
--builtin_ntp_servers=127.25.109.148:38401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:53:18.191004 30850 heartbeater.cc:347] Connected to a master server at 127.25.109.190:35257
I20240502 01:53:18.191377 30850 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:18.192356 30850 heartbeater.cc:511] Master 127.25.109.190:35257 requested a full tablet report, sending...
I20240502 01:53:18.194350 30524 ts_manager.cc:194] Registered new tserver with Master: ab67584cb8b5451f9575a56d3593f032 (127.25.109.130:43749)
I20240502 01:53:18.195313 30524 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.130:55717
W20240502 01:53:18.519450 30854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:18.519802 30854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:18.519965 30854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:18.527505 30854 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:53:18.527664 30854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:18.527841 30854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.131
I20240502 01:53:18.537752 30854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:38401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.25.109.131
--webserver_port=0
--tserver_master_addrs=127.25.109.190:35257
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.131
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:18.539094 30854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:18.540912 30854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:18.553050 30860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:18.553565 30862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:18.554004 30859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:18.554641 30854 server_base.cc:1034] running on GCE node
I20240502 01:53:18.555259 30854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:18.556316 30854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:18.557521 30854 hybrid_clock.cc:648] HybridClock initialized: now 1714614798557441 us; error 76 us; skew 500 ppm
I20240502 01:53:18.557848 30854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:18.560132 30854 webserver.cc:460] Webserver started at http://127.25.109.131:34413/ using document root <none> and password file <none>
I20240502 01:53:18.560889 30854 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:18.561000 30854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:18.561305 30854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:18.566383 30854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "7b100a3dd2a34d5d97da5b1098823c3c"
format_stamp: "Formatted at 2024-05-02 01:53:18 on dist-test-slave-kj4b"
I20240502 01:53:18.567158 30854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "7b100a3dd2a34d5d97da5b1098823c3c"
format_stamp: "Formatted at 2024-05-02 01:53:18 on dist-test-slave-kj4b"
I20240502 01:53:18.572490 30854 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.002s
I20240502 01:53:18.576072 30868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:18.577509 30854 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240502 01:53:18.577685 30854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "7b100a3dd2a34d5d97da5b1098823c3c"
format_stamp: "Formatted at 2024-05-02 01:53:18 on dist-test-slave-kj4b"
I20240502 01:53:18.577896 30854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:18.592222 30854 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:53:18.593101 30854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:18.593369 30854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:18.594311 30854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:18.595789 30854 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:18.595927 30854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:18.596029 30854 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:18.596097 30854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:18.625808 30854 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.131:37667
I20240502 01:53:18.625883 30981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.131:37667 every 8 connection(s)
I20240502 01:53:18.627113 30854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 01:53:18.635707 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 30854
I20240502 01:53:18.635937 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 01:53:18.639465 30982 heartbeater.cc:347] Connected to a master server at 127.25.109.190:35257
I20240502 01:53:18.639483 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.132:0
--local_ip_for_outbound_sockets=127.25.109.132
--webserver_interface=127.25.109.132
--webserver_port=0
--tserver_master_addrs=127.25.109.190:35257
--builtin_ntp_servers=127.25.109.148:38401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:53:18.639788 30982 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:18.640559 30982 heartbeater.cc:511] Master 127.25.109.190:35257 requested a full tablet report, sending...
I20240502 01:53:18.641927 30524 ts_manager.cc:194] Registered new tserver with Master: 7b100a3dd2a34d5d97da5b1098823c3c (127.25.109.131:37667)
I20240502 01:53:18.642714 30524 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.131:57875
I20240502 01:53:18.744619 30715 heartbeater.cc:503] Master 127.25.109.190:35257 was elected leader, sending a full tablet report...
W20240502 01:53:18.961808 30985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:18.962121 30985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:18.962251 30985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:18.969894 30985 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:53:18.970052 30985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:18.970203 30985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.132
I20240502 01:53:18.979466 30985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:38401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.25.109.132
--webserver_port=0
--tserver_master_addrs=127.25.109.190:35257
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.132
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:18.980823 30985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:18.982658 30985 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:18.994951 30990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:18.995333 30991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:18.995947 30994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:18.996429 30985 server_base.cc:1034] running on GCE node
I20240502 01:53:18.997040 30985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:18.998016 30985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:18.999209 30985 hybrid_clock.cc:648] HybridClock initialized: now 1714614798999155 us; error 53 us; skew 500 ppm
I20240502 01:53:18.999517 30985 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:19.001695 30985 webserver.cc:460] Webserver started at http://127.25.109.132:43551/ using document root <none> and password file <none>
I20240502 01:53:19.002403 30985 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:19.002501 30985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:19.002777 30985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:19.005530 30985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "f7f47d3f7da1479ea081d26462b3a31e"
format_stamp: "Formatted at 2024-05-02 01:53:18 on dist-test-slave-kj4b"
I20240502 01:53:19.006269 30985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "f7f47d3f7da1479ea081d26462b3a31e"
format_stamp: "Formatted at 2024-05-02 01:53:18 on dist-test-slave-kj4b"
I20240502 01:53:19.011538 30985 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20240502 01:53:19.014963 31000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:19.016790 30985 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.003s
I20240502 01:53:19.016969 30985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "f7f47d3f7da1479ea081d26462b3a31e"
format_stamp: "Formatted at 2024-05-02 01:53:18 on dist-test-slave-kj4b"
I20240502 01:53:19.017174 30985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:19.048349 30985 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:53:19.049542 30985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:19.049890 30985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:19.051156 30985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:19.052939 30985 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:19.053025 30985 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:19.053112 30985 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:19.053162 30985 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:19.083191 30985 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.132:40195
I20240502 01:53:19.083276 31114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.132:40195 every 8 connection(s)
I20240502 01:53:19.084563 30985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 01:53:19.087374 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 30985
I20240502 01:53:19.087574 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 01:53:19.090817 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.133:0
--local_ip_for_outbound_sockets=127.25.109.133
--webserver_interface=127.25.109.133
--webserver_port=0
--tserver_master_addrs=127.25.109.190:35257
--builtin_ntp_servers=127.25.109.148:38401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:53:19.097980 31115 heartbeater.cc:347] Connected to a master server at 127.25.109.190:35257
I20240502 01:53:19.098320 31115 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:19.099246 31115 heartbeater.cc:511] Master 127.25.109.190:35257 requested a full tablet report, sending...
I20240502 01:53:19.100862 30524 ts_manager.cc:194] Registered new tserver with Master: f7f47d3f7da1479ea081d26462b3a31e (127.25.109.132:40195)
I20240502 01:53:19.101670 30524 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.132:34605
I20240502 01:53:19.198582 30850 heartbeater.cc:503] Master 127.25.109.190:35257 was elected leader, sending a full tablet report...
W20240502 01:53:19.410275 31118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:19.410580 31118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:19.410698 31118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:19.418205 31118 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:53:19.418378 31118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:19.418510 31118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.133
I20240502 01:53:19.427971 31118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:38401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.25.109.133
--webserver_port=0
--tserver_master_addrs=127.25.109.190:35257
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.133
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:19.429334 31118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:19.431044 31118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:19.442932 31124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:19.443509 31126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:19.444072 31123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:19.444625 31118 server_base.cc:1034] running on GCE node
I20240502 01:53:19.445187 31118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:19.446115 31118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:19.447317 31118 hybrid_clock.cc:648] HybridClock initialized: now 1714614799447268 us; error 44 us; skew 500 ppm
I20240502 01:53:19.447628 31118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:19.449908 31118 webserver.cc:460] Webserver started at http://127.25.109.133:45471/ using document root <none> and password file <none>
I20240502 01:53:19.450654 31118 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:19.450776 31118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:19.451093 31118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:19.453804 31118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "8321a814e5fb4a7bb644199438996680"
format_stamp: "Formatted at 2024-05-02 01:53:19 on dist-test-slave-kj4b"
I20240502 01:53:19.454536 31118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "8321a814e5fb4a7bb644199438996680"
format_stamp: "Formatted at 2024-05-02 01:53:19 on dist-test-slave-kj4b"
I20240502 01:53:19.460176 31118 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.001s
I20240502 01:53:19.463587 31132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:19.465039 31118 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 01:53:19.465215 31118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "8321a814e5fb4a7bb644199438996680"
format_stamp: "Formatted at 2024-05-02 01:53:19 on dist-test-slave-kj4b"
I20240502 01:53:19.465420 31118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:19.486019 31118 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:53:19.486927 31118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:19.487201 31118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:19.488260 31118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:19.489814 31118 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:19.489908 31118 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:19.490016 31118 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:19.490077 31118 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:19.518576 31118 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.133:42873
I20240502 01:53:19.518644 31245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.133:42873 every 8 connection(s)
I20240502 01:53:19.519985 31118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:53:19.520298 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 31118
I20240502 01:53:19.520453 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 01:53:19.533507 31246 heartbeater.cc:347] Connected to a master server at 127.25.109.190:35257
I20240502 01:53:19.533812 31246 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:19.534619 31246 heartbeater.cc:511] Master 127.25.109.190:35257 requested a full tablet report, sending...
I20240502 01:53:19.536068 30524 ts_manager.cc:194] Registered new tserver with Master: 8321a814e5fb4a7bb644199438996680 (127.25.109.133:42873)
I20240502 01:53:19.536676 26038 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 01:53:19.537004 30524 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.133:33297
I20240502 01:53:19.553303 30524 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46396:
name: "TestTable"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
owner: "alice"
I20240502 01:53:19.586503 30783 tablet_service.cc:1467] Processing CreateTablet for tablet 84746e0f662e4a9587b1984509d32ada (DEFAULT_TABLE table=TestTable [id=6305b124ce6047ec9295d69fb23c8040]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:19.588294 30783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84746e0f662e4a9587b1984509d32ada. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:19.594611 31049 tablet_service.cc:1467] Processing CreateTablet for tablet 84746e0f662e4a9587b1984509d32ada (DEFAULT_TABLE table=TestTable [id=6305b124ce6047ec9295d69fb23c8040]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:19.596280 31049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84746e0f662e4a9587b1984509d32ada. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:19.597553 31180 tablet_service.cc:1467] Processing CreateTablet for tablet 84746e0f662e4a9587b1984509d32ada (DEFAULT_TABLE table=TestTable [id=6305b124ce6047ec9295d69fb23c8040]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:19.598829 31180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84746e0f662e4a9587b1984509d32ada. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:19.602458 31264 tablet_bootstrap.cc:492] T 84746e0f662e4a9587b1984509d32ada P ab67584cb8b5451f9575a56d3593f032: Bootstrap starting.
I20240502 01:53:19.606942 31264 tablet_bootstrap.cc:654] T 84746e0f662e4a9587b1984509d32ada P ab67584cb8b5451f9575a56d3593f032: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:19.607815 31265 tablet_bootstrap.cc:492] T 84746e0f662e4a9587b1984509d32ada P f7f47d3f7da1479ea081d26462b3a31e: Bootstrap starting.
I20240502 01:53:19.608333 31264 log.cc:824] T 84746e0f662e4a9587b1984509d32ada P ab67584cb8b5451f9575a56d3593f032: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:19.611380 31264 tablet_bootstrap.cc:492] T 84746e0f662e4a9587b1984509d32ada P ab67584cb8b5451f9575a56d3593f032: No bootstrap required, opened a new log
I20240502 01:53:19.611369 31267 tablet_bootstrap.cc:492] T 84746e0f662e4a9587b1984509d32ada P 8321a814e5fb4a7bb644199438996680: Bootstrap starting.
I20240502 01:53:19.611788 31264 ts_tablet_manager.cc:1367] T 84746e0f662e4a9587b1984509d32ada P ab67584cb8b5451f9575a56d3593f032: Time spent bootstrapping tablet: real 0.010s	user 0.008s	sys 0.000s
I20240502 01:53:19.612429 31265 tablet_bootstrap.cc:654] T 84746e0f662e4a9587b1984509d32ada P f7f47d3f7da1479ea081d26462b3a31e: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:19.613621 31265 log.cc:824] T 84746e0f662e4a9587b1984509d32ada P f7f47d3f7da1479ea081d26462b3a31e: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:19.614652 31267 tablet_bootstrap.cc:654] T 84746e0f662e4a9587b1984509d32ada P 8321a814e5fb4a7bb644199438996680: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:19.615792 31267 log.cc:824] T 84746e0f662e4a9587b1984509d32ada P 8321a814e5fb4a7bb644199438996680: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:19.616695 31265 tablet_bootstrap.cc:492] T 84746e0f662e4a9587b1984509d32ada P f7f47d3f7da1479ea081d26462b3a31e: No bootstrap required, opened a new log
I20240502 01:53:19.616950 31265 ts_tablet_manager.cc:1367] T 84746e0f662e4a9587b1984509d32ada P f7f47d3f7da1479ea081d26462b3a31e: Time spent bootstrapping tablet: real 0.009s	user 0.007s	sys 0.000s
I20240502 01:53:19.618646 31267 tablet_bootstrap.cc:492] T 84746e0f662e4a9587b1984509d32ada P 8321a814e5fb4a7bb644199438996680: No bootstrap required, opened a new log
I20240502 01:53:19.618922 31267 ts_tablet_manager.cc:1367] T 84746e0f662e4a9587b1984509d32ada P 8321a814e5fb4a7bb644199438996680: Time spent bootstrapping tablet: real 0.008s	user 0.003s	sys 0.003s
I20240502 01:53:19.619184 31264 raft_consensus.cc:348] T 84746e0f662e4a9587b1984509d32ada P ab67584cb8b5451f9575a56d3593f032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab67584cb8b5451f9575a56d3593f032" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43749 } } peers { permanent_uuid: "f7f47d3f7da1479ea081d26462b3a31e" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 40195 } } peers { permanent_uuid: "8321a814e5fb4a7bb644199438996680" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 42873 } }
I20240502 01:53:19.619647 31264 raft_consensus.cc:374] T 84746e0f662e4a9587b1984509d32ada P ab67584cb8b5451f9575a56d3593f032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:19.619796 31264 raft_consensus.cc:729] T 84746e0f662e4a9587b1984509d32ada P ab67584cb8b5451f9575a56d3593f032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab67584cb8b5451f9575a56d3593f032, State: Initialized, Role: FOLLOWER
I20240502 01:53:19.620574 31264 consensus_queue.cc:260] T 84746e0f662e4a9587b1984509d32ada P ab67584cb8b5451f9575a56d3593f032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab67584cb8b5451f9575a56d3593f032" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43749 } } peers { permanent_uuid: "f7f47d3f7da1479ea081d26462b3a31e" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 40195 } } peers { permanent_uuid: "8321a814e5fb4a7bb644199438996680" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 42873 } }
I20240502 01:53:19.622306 31264 ts_tablet_manager.cc:1398] T 84746e0f662e4a9587b1984509d32ada P ab67584cb8b5451f9575a56d3593f032: Time spent starting tablet: real 0.010s	user 0.009s	sys 0.000s
I20240502 01:53:19.625774 31265 raft_consensus.cc:348] T 84746e0f662e4a9587b1984509d32ada P f7f47d3f7da1479ea081d26462b3a31e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab67584cb8b5451f9575a56d3593f032" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43749 } } peers { permanent_uuid: "f7f47d3f7da1479ea081d26462b3a31e" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 40195 } } peers { permanent_uuid: "8321a814e5fb4a7bb644199438996680" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 42873 } }
I20240502 01:53:19.626184 31265 raft_consensus.cc:374] T 84746e0f662e4a9587b1984509d32ada P f7f47d3f7da1479ea081d26462b3a31e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:19.626283 31265 raft_consensus.cc:729] T 84746e0f662e4a9587b1984509d32ada P f7f47d3f7da1479ea081d26462b3a31e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7f47d3f7da1479ea081d26462b3a31e, State: Initialized, Role: FOLLOWER
I20240502 01:53:19.626878 31265 consensus_queue.cc:260] T 84746e0f662e4a9587b1984509d32ada P f7f47d3f7da1479ea081d26462b3a31e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab67584cb8b5451f9575a56d3593f032" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43749 } } peers { permanent_uuid: "f7f47d3f7da1479ea081d26462b3a31e" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 40195 } } peers { permanent_uuid: "8321a814e5fb4a7bb644199438996680" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 42873 } }
I20240502 01:53:19.626701 31267 raft_consensus.cc:348] T 84746e0f662e4a9587b1984509d32ada P 8321a814e5fb4a7bb644199438996680 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab67584cb8b5451f9575a56d3593f032" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43749 } } peers { permanent_uuid: "f7f47d3f7da1479ea081d26462b3a31e" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 40195 } } peers { permanent_uuid: "8321a814e5fb4a7bb644199438996680" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 42873 } }
I20240502 01:53:19.627112 31267 raft_consensus.cc:374] T 84746e0f662e4a9587b1984509d32ada P 8321a814e5fb4a7bb644199438996680 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:19.627228 31267 raft_consensus.cc:729] T 84746e0f662e4a9587b1984509d32ada P 8321a814e5fb4a7bb644199438996680 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8321a814e5fb4a7bb644199438996680, State: Initialized, Role: FOLLOWER
I20240502 01:53:19.627961 31267 consensus_queue.cc:260] T 84746e0f662e4a9587b1984509d32ada P 8321a814e5fb4a7bb644199438996680 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab67584cb8b5451f9575a56d3593f032" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43749 } } peers { permanent_uuid: "f7f47d3f7da1479ea081d26462b3a31e" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 40195 } } peers { permanent_uuid: "8321a814e5fb4a7bb644199438996680" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 42873 } }
I20240502 01:53:19.629350 31246 heartbeater.cc:503] Master 127.25.109.190:35257 was elected leader, sending a full tablet report...
I20240502 01:53:19.629791 31267 ts_tablet_manager.cc:1398] T 84746e0f662e4a9587b1984509d32ada P 8321a814e5fb4a7bb644199438996680: Time spent starting tablet: real 0.011s	user 0.011s	sys 0.000s
I20240502 01:53:19.631017 31115 heartbeater.cc:503] Master 127.25.109.190:35257 was elected leader, sending a full tablet report...
I20240502 01:53:19.631538 31265 ts_tablet_manager.cc:1398] T 84746e0f662e4a9587b1984509d32ada P f7f47d3f7da1479ea081d26462b3a31e: Time spent starting tablet: real 0.014s	user 0.012s	sys 0.000s
I20240502 01:53:19.645769 30982 heartbeater.cc:503] Master 127.25.109.190:35257 was elected leader, sending a full tablet report...
I20240502 01:53:19.651225 31272 raft_consensus.cc:482] T 84746e0f662e4a9587b1984509d32ada P 8321a814e5fb4a7bb644199438996680 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:19.651530 31272 raft_consensus.cc:504] T 84746e0f662e4a9587b1984509d32ada P 8321a814e5fb4a7bb644199438996680 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab67584cb8b5451f9575a56d3593f032" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43749 } } peers { permanent_uuid: "f7f47d3f7da1479ea081d26462b3a31e" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 40195 } } peers { permanent_uuid: "8321a814e5fb4a7bb644199438996680" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 42873 } }
I20240502 01:53:19.653276 31272 leader_election.cc:288] T 84746e0f662e4a9587b1984509d32ada P 8321a814e5fb4a7bb644199438996680 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab67584cb8b5451f9575a56d3593f032 (127.25.109.130:43749), f7f47d3f7da1479ea081d26462b3a31e (127.25.109.132:40195)
I20240502 01:53:19.659776 30803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84746e0f662e4a9587b1984509d32ada" candidate_uuid: "8321a814e5fb4a7bb644199438996680" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab67584cb8b5451f9575a56d3593f032" is_pre_election: true
I20240502 01:53:19.660396 30803 raft_consensus.cc:2454] T 84746e0f662e4a9587b1984509d32ada P ab67584cb8b5451f9575a56d3593f032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8321a814e5fb4a7bb644199438996680 in term 0.
I20240502 01:53:19.660439 31069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84746e0f662e4a9587b1984509d32ada" candidate_uuid: "8321a814e5fb4a7bb644199438996680" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7f47d3f7da1479ea081d26462b3a31e" is_pre_election: true
I20240502 01:53:19.661001 31069 raft_consensus.cc:2454] T 84746e0f662e4a9587b1984509d32ada P f7f47d3f7da1479ea081d26462b3a31e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8321a814e5fb4a7bb644199438996680 in term 0.
I20240502 01:53:19.661192 31135 leader_election.cc:302] T 84746e0f662e4a9587b1984509d32ada P 8321a814e5fb4a7bb644199438996680 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8321a814e5fb4a7bb644199438996680, ab67584cb8b5451f9575a56d3593f032; no voters: 
I20240502 01:53:19.661597 31272 raft_consensus.cc:2781] T 84746e0f662e4a9587b1984509d32ada P 8321a814e5fb4a7bb644199438996680 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:19.661733 31272 raft_consensus.cc:482] T 84746e0f662e4a9587b1984509d32ada P 8321a814e5fb4a7bb644199438996680 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:19.661844 31272 raft_consensus.cc:3037] T 84746e0f662e4a9587b1984509d32ada P 8321a814e5fb4a7bb644199438996680 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:19.663890 31272 raft_consensus.cc:504] T 84746e0f662e4a9587b1984509d32ada P 8321a814e5fb4a7bb644199438996680 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab67584cb8b5451f9575a56d3593f032" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43749 } } peers { permanent_uuid: "f7f47d3f7da1479ea081d26462b3a31e" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 40195 } } peers { permanent_uuid: "8321a814e5fb4a7bb644199438996680" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 42873 } }
I20240502 01:53:19.664477 31272 leader_election.cc:288] T 84746e0f662e4a9587b1984509d32ada P 8321a814e5fb4a7bb644199438996680 [CANDIDATE]: Term 1 election: Requested vote from peers ab67584cb8b5451f9575a56d3593f032 (127.25.109.130:43749), f7f47d3f7da1479ea081d26462b3a31e (127.25.109.132:40195)
I20240502 01:53:19.665043 30803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84746e0f662e4a9587b1984509d32ada" candidate_uuid: "8321a814e5fb4a7bb644199438996680" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab67584cb8b5451f9575a56d3593f032"
I20240502 01:53:19.665064 31069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84746e0f662e4a9587b1984509d32ada" candidate_uuid: "8321a814e5fb4a7bb644199438996680" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7f47d3f7da1479ea081d26462b3a31e"
I20240502 01:53:19.665227 31069 raft_consensus.cc:3037] T 84746e0f662e4a9587b1984509d32ada P f7f47d3f7da1479ea081d26462b3a31e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:19.665242 30803 raft_consensus.cc:3037] T 84746e0f662e4a9587b1984509d32ada P ab67584cb8b5451f9575a56d3593f032 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:19.667294 31069 raft_consensus.cc:2454] T 84746e0f662e4a9587b1984509d32ada P f7f47d3f7da1479ea081d26462b3a31e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8321a814e5fb4a7bb644199438996680 in term 1.
I20240502 01:53:19.667315 30803 raft_consensus.cc:2454] T 84746e0f662e4a9587b1984509d32ada P ab67584cb8b5451f9575a56d3593f032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8321a814e5fb4a7bb644199438996680 in term 1.
I20240502 01:53:19.667709 31135 leader_election.cc:302] T 84746e0f662e4a9587b1984509d32ada P 8321a814e5fb4a7bb644199438996680 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8321a814e5fb4a7bb644199438996680, f7f47d3f7da1479ea081d26462b3a31e; no voters: 
I20240502 01:53:19.668069 31272 raft_consensus.cc:2781] T 84746e0f662e4a9587b1984509d32ada P 8321a814e5fb4a7bb644199438996680 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:19.668488 31272 raft_consensus.cc:686] T 84746e0f662e4a9587b1984509d32ada P 8321a814e5fb4a7bb644199438996680 [term 1 LEADER]: Becoming Leader. State: Replica: 8321a814e5fb4a7bb644199438996680, State: Running, Role: LEADER
I20240502 01:53:19.669039 31272 consensus_queue.cc:237] T 84746e0f662e4a9587b1984509d32ada P 8321a814e5fb4a7bb644199438996680 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab67584cb8b5451f9575a56d3593f032" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43749 } } peers { permanent_uuid: "f7f47d3f7da1479ea081d26462b3a31e" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 40195 } } peers { permanent_uuid: "8321a814e5fb4a7bb644199438996680" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 42873 } }
I20240502 01:53:19.673607 30525 catalog_manager.cc:5484] T 84746e0f662e4a9587b1984509d32ada P 8321a814e5fb4a7bb644199438996680 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8321a814e5fb4a7bb644199438996680 (127.25.109.133). New cstate: current_term: 1 leader_uuid: "8321a814e5fb4a7bb644199438996680" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab67584cb8b5451f9575a56d3593f032" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43749 } } peers { permanent_uuid: "f7f47d3f7da1479ea081d26462b3a31e" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 40195 } } peers { permanent_uuid: "8321a814e5fb4a7bb644199438996680" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 42873 } } }
W20240502 01:53:19.675177 30852 tablet.cc:2369] T 84746e0f662e4a9587b1984509d32ada P ab67584cb8b5451f9575a56d3593f032: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:53:19.703495 26038 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 01:53:19.706331 26038 ts_itest-base.cc:233] Waiting for 0 tablets on tserver d96d46cf42454fb9ba48f87dc8d8dd2f to finish bootstrapping
I20240502 01:53:19.715947 26038 ts_itest-base.cc:233] Waiting for 1 tablets on tserver ab67584cb8b5451f9575a56d3593f032 to finish bootstrapping
I20240502 01:53:19.724949 26038 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 7b100a3dd2a34d5d97da5b1098823c3c to finish bootstrapping
I20240502 01:53:19.733836 26038 ts_itest-base.cc:233] Waiting for 1 tablets on tserver f7f47d3f7da1479ea081d26462b3a31e to finish bootstrapping
I20240502 01:53:19.741449 26038 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 8321a814e5fb4a7bb644199438996680 to finish bootstrapping
W20240502 01:53:19.773135 31247 tablet.cc:2369] T 84746e0f662e4a9587b1984509d32ada P 8321a814e5fb4a7bb644199438996680: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:53:19.836567 31116 tablet.cc:2369] T 84746e0f662e4a9587b1984509d32ada P f7f47d3f7da1479ea081d26462b3a31e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:53:20.156728 31272 consensus_queue.cc:1035] T 84746e0f662e4a9587b1984509d32ada P 8321a814e5fb4a7bb644199438996680 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f47d3f7da1479ea081d26462b3a31e" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:20.168316 31272 consensus_queue.cc:1035] T 84746e0f662e4a9587b1984509d32ada P 8321a814e5fb4a7bb644199438996680 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab67584cb8b5451f9575a56d3593f032" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:20.213562 31180 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:20.214975 30916 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:20.215970 30783 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:20.218968 30649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:20.245045 31049 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:20.386866 30916 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:20.388082 30783 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:20.393400 31180 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:20.398164 30649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:20.410668 31049 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:20.985095 30916 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:20.985139 31180 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:20.990355 30649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:20.991004 30783 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:20.993273 31049 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:21.150259 30916 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:21.151413 31180 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:21.155274 30649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:21.157308 30783 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:21.182502 31049 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:21.350631 31180 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:21.350836 30916 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:21.354460 30783 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:21.361145 30649 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:21.378674 31049 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:21.516961 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 30586
I20240502 01:53:21.518466 30710 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:21.618664 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 30586
I20240502 01:53:21.638410 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 30720
I20240502 01:53:21.639482 30845 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:21.735018 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 30720
I20240502 01:53:21.751221 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 30854
I20240502 01:53:21.752331 30976 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:21.842445 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 30854
I20240502 01:53:21.857717 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 30985
I20240502 01:53:21.858767 31109 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 01:53:21.872344 31135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.130:43749: connect: Connection refused (error 111)
W20240502 01:53:21.873895 31135 consensus_peers.cc:485] T 84746e0f662e4a9587b1984509d32ada P 8321a814e5fb4a7bb644199438996680 -> Peer ab67584cb8b5451f9575a56d3593f032 (127.25.109.130:43749): Couldn't send request to peer ab67584cb8b5451f9575a56d3593f032. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.130:43749: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:21.954640 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 30985
I20240502 01:53:21.970118 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 31118
I20240502 01:53:21.971159 31240 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:22.066118 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 31118
I20240502 01:53:22.081087 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 30494
I20240502 01:53:22.082356 30557 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:22.199406 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 30494
2024-05-02T01:53:22Z chronyd exiting
[       OK ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/0 (5544 ms)
[----------] 1 test from RebalanceStartCriteriaTest (5544 ms total)

[----------] 1 test from RebalanceParamTest
[ RUN      ] RebalanceParamTest.Rebalance/2
I20240502 01:53:22.241875 26038 test_util.cc:268] Using random seed: -566830806
I20240502 01:53:22.243803 26038 ts_itest-base.cc:112] Starting cluster with:
I20240502 01:53:22.243911 26038 ts_itest-base.cc:113] --------------
I20240502 01:53:22.243986 26038 ts_itest-base.cc:114] 7 tablet servers
I20240502 01:53:22.244032 26038 ts_itest-base.cc:115] 2 replicas per TS
I20240502 01:53:22.244096 26038 ts_itest-base.cc:116] --------------
2024-05-02T01:53:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T01:53:22Z Disabled control of system clock
I20240502 01:53:22.269239 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.190:41417
--webserver_interface=127.25.109.190
--webserver_port=0
--builtin_ntp_servers=127.25.109.148:40813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.190:41417
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240502 01:53:22.605759 31474 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 01:53:22.606125 31474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:22.606225 31474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:22.606298 31474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:22.614203 31474 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:53:22.614329 31474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 01:53:22.614396 31474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:22.614444 31474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 01:53:22.614503 31474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 01:53:22.623978 31474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:40813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.25.109.190:41417
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.190:41417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.25.109.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:22.625300 31474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:22.627086 31474 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:22.638073 31479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:22.638607 31480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:22.638998 31482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:22.639647 31474 server_base.cc:1034] running on GCE node
I20240502 01:53:22.640499 31474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:22.642042 31474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:22.643298 31474 hybrid_clock.cc:648] HybridClock initialized: now 1714614802643209 us; error 53 us; skew 500 ppm
I20240502 01:53:22.643651 31474 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:22.659492 31474 webserver.cc:460] Webserver started at http://127.25.109.190:40347/ using document root <none> and password file <none>
I20240502 01:53:22.660338 31474 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:22.660483 31474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:22.660794 31474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:22.663537 31474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "15725d2cffce4f5da0ed6c7449588f85"
format_stamp: "Formatted at 2024-05-02 01:53:22 on dist-test-slave-kj4b"
I20240502 01:53:22.664315 31474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "15725d2cffce4f5da0ed6c7449588f85"
format_stamp: "Formatted at 2024-05-02 01:53:22 on dist-test-slave-kj4b"
I20240502 01:53:22.669677 31474 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20240502 01:53:22.673261 31488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:22.674614 31474 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240502 01:53:22.674798 31474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal
uuid: "15725d2cffce4f5da0ed6c7449588f85"
format_stamp: "Formatted at 2024-05-02 01:53:22 on dist-test-slave-kj4b"
I20240502 01:53:22.674998 31474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:22.700042 31474 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:53:22.700966 31474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:22.701287 31474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:22.719532 31474 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.190:41417
I20240502 01:53:22.719550 31539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.190:41417 every 8 connection(s)
I20240502 01:53:22.721107 31474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 01:53:22.725260 31540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:22.728790 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 31474
I20240502 01:53:22.729023 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 01:53:22.736855 31540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15725d2cffce4f5da0ed6c7449588f85: Bootstrap starting.
I20240502 01:53:22.740818 31540 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15725d2cffce4f5da0ed6c7449588f85: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:22.742287 31540 log.cc:824] T 00000000000000000000000000000000 P 15725d2cffce4f5da0ed6c7449588f85: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:22.745260 31540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15725d2cffce4f5da0ed6c7449588f85: No bootstrap required, opened a new log
I20240502 01:53:22.751600 31540 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15725d2cffce4f5da0ed6c7449588f85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15725d2cffce4f5da0ed6c7449588f85" member_type: VOTER last_known_addr { host: "127.25.109.190" port: 41417 } }
I20240502 01:53:22.752106 31540 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15725d2cffce4f5da0ed6c7449588f85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:22.752255 31540 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15725d2cffce4f5da0ed6c7449588f85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15725d2cffce4f5da0ed6c7449588f85, State: Initialized, Role: FOLLOWER
I20240502 01:53:22.752966 31540 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15725d2cffce4f5da0ed6c7449588f85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15725d2cffce4f5da0ed6c7449588f85" member_type: VOTER last_known_addr { host: "127.25.109.190" port: 41417 } }
I20240502 01:53:22.753190 31540 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15725d2cffce4f5da0ed6c7449588f85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:53:22.753327 31540 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15725d2cffce4f5da0ed6c7449588f85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:53:22.753496 31540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15725d2cffce4f5da0ed6c7449588f85 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:22.755640 31540 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15725d2cffce4f5da0ed6c7449588f85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15725d2cffce4f5da0ed6c7449588f85" member_type: VOTER last_known_addr { host: "127.25.109.190" port: 41417 } }
I20240502 01:53:22.756196 31540 leader_election.cc:302] T 00000000000000000000000000000000 P 15725d2cffce4f5da0ed6c7449588f85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15725d2cffce4f5da0ed6c7449588f85; no voters: 
I20240502 01:53:22.756860 31540 leader_election.cc:288] T 00000000000000000000000000000000 P 15725d2cffce4f5da0ed6c7449588f85 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 01:53:22.757048 31545 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15725d2cffce4f5da0ed6c7449588f85 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:22.757696 31545 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15725d2cffce4f5da0ed6c7449588f85 [term 1 LEADER]: Becoming Leader. State: Replica: 15725d2cffce4f5da0ed6c7449588f85, State: Running, Role: LEADER
I20240502 01:53:22.758332 31545 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15725d2cffce4f5da0ed6c7449588f85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15725d2cffce4f5da0ed6c7449588f85" member_type: VOTER last_known_addr { host: "127.25.109.190" port: 41417 } }
I20240502 01:53:22.759617 31540 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15725d2cffce4f5da0ed6c7449588f85 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:53:22.761166 31547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15725d2cffce4f5da0ed6c7449588f85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15725d2cffce4f5da0ed6c7449588f85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15725d2cffce4f5da0ed6c7449588f85" member_type: VOTER last_known_addr { host: "127.25.109.190" port: 41417 } } }
I20240502 01:53:22.761513 31547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15725d2cffce4f5da0ed6c7449588f85 [sys.catalog]: This master's current role is: LEADER
I20240502 01:53:22.761191 31546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15725d2cffce4f5da0ed6c7449588f85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15725d2cffce4f5da0ed6c7449588f85. Latest consensus state: current_term: 1 leader_uuid: "15725d2cffce4f5da0ed6c7449588f85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15725d2cffce4f5da0ed6c7449588f85" member_type: VOTER last_known_addr { host: "127.25.109.190" port: 41417 } } }
I20240502 01:53:22.762739 31546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15725d2cffce4f5da0ed6c7449588f85 [sys.catalog]: This master's current role is: LEADER
I20240502 01:53:22.764623 31550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:53:22.773334 31550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:53:22.784068 31550 catalog_manager.cc:1347] Generated new cluster ID: 1437b79ff61a4174a4c22d91c35b9236
I20240502 01:53:22.784214 31550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:53:22.800513 31550 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:53:22.801813 31550 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:53:22.814138 31550 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 15725d2cffce4f5da0ed6c7449588f85: Generated new TSK 0
I20240502 01:53:22.815026 31550 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:53:22.822757 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.129:0
--local_ip_for_outbound_sockets=127.25.109.129
--webserver_interface=127.25.109.129
--webserver_port=0
--tserver_master_addrs=127.25.109.190:41417
--builtin_ntp_servers=127.25.109.148:40813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 01:53:23.155179 31565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:23.155501 31565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:23.155618 31565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:23.163425 31565 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:53:23.163569 31565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:23.163725 31565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.129
I20240502 01:53:23.173244 31565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:40813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.25.109.129
--webserver_port=0
--tserver_master_addrs=127.25.109.190:41417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.129
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:23.174643 31565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:23.176527 31565 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:23.189553 31573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:23.190013 31571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:23.190474 31570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:23.191166 31565 server_base.cc:1034] running on GCE node
I20240502 01:53:23.191841 31565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:23.192925 31565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:23.194155 31565 hybrid_clock.cc:648] HybridClock initialized: now 1714614803194096 us; error 47 us; skew 500 ppm
I20240502 01:53:23.194519 31565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:23.196872 31565 webserver.cc:460] Webserver started at http://127.25.109.129:42465/ using document root <none> and password file <none>
I20240502 01:53:23.197651 31565 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:23.197798 31565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:23.198130 31565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:23.200987 31565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "b191086bf5f2432b8d240510817ae5fe"
format_stamp: "Formatted at 2024-05-02 01:53:23 on dist-test-slave-kj4b"
I20240502 01:53:23.201824 31565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "b191086bf5f2432b8d240510817ae5fe"
format_stamp: "Formatted at 2024-05-02 01:53:23 on dist-test-slave-kj4b"
I20240502 01:53:23.207515 31565 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.001s
I20240502 01:53:23.211231 31580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:23.212769 31565 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 01:53:23.212942 31565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b191086bf5f2432b8d240510817ae5fe"
format_stamp: "Formatted at 2024-05-02 01:53:23 on dist-test-slave-kj4b"
I20240502 01:53:23.213153 31565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:23.245359 31565 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:53:23.246255 31565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:23.246541 31565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:23.247491 31565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:23.249086 31565 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:23.249188 31565 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:23.249301 31565 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:23.249372 31565 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:23.279852 31565 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.129:42243
I20240502 01:53:23.279959 31696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.129:42243 every 8 connection(s)
I20240502 01:53:23.281266 31565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 01:53:23.291625 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 31565
I20240502 01:53:23.291900 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 01:53:23.294910 31697 heartbeater.cc:347] Connected to a master server at 127.25.109.190:41417
I20240502 01:53:23.295253 31697 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:23.295642 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.130:0
--local_ip_for_outbound_sockets=127.25.109.130
--webserver_interface=127.25.109.130
--webserver_port=0
--tserver_master_addrs=127.25.109.190:41417
--builtin_ntp_servers=127.25.109.148:40813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:53:23.296144 31697 heartbeater.cc:511] Master 127.25.109.190:41417 requested a full tablet report, sending...
I20240502 01:53:23.298235 31505 ts_manager.cc:194] Registered new tserver with Master: b191086bf5f2432b8d240510817ae5fe (127.25.109.129:42243)
I20240502 01:53:23.299769 31505 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.129:41959
W20240502 01:53:23.627810 31701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:23.628170 31701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:23.628271 31701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:23.635823 31701 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:53:23.636005 31701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:23.636138 31701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.130
I20240502 01:53:23.645541 31701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:40813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.25.109.130
--webserver_port=0
--tserver_master_addrs=127.25.109.190:41417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.130
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:23.646970 31701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:23.648757 31701 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:23.661032 31706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:23.661576 31707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:23.662178 31709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:23.662766 31701 server_base.cc:1034] running on GCE node
I20240502 01:53:23.663380 31701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:23.664410 31701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:23.665640 31701 hybrid_clock.cc:648] HybridClock initialized: now 1714614803665585 us; error 49 us; skew 500 ppm
I20240502 01:53:23.665995 31701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:23.668349 31701 webserver.cc:460] Webserver started at http://127.25.109.130:43285/ using document root <none> and password file <none>
I20240502 01:53:23.669111 31701 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:23.669222 31701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:23.669534 31701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:23.673807 31701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "35f356f20e274db7bf020f78f1bc9ae7"
format_stamp: "Formatted at 2024-05-02 01:53:23 on dist-test-slave-kj4b"
I20240502 01:53:23.674585 31701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "35f356f20e274db7bf020f78f1bc9ae7"
format_stamp: "Formatted at 2024-05-02 01:53:23 on dist-test-slave-kj4b"
I20240502 01:53:23.679975 31701 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.001s
I20240502 01:53:23.683674 31715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:23.685284 31701 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.002s
I20240502 01:53:23.685485 31701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "35f356f20e274db7bf020f78f1bc9ae7"
format_stamp: "Formatted at 2024-05-02 01:53:23 on dist-test-slave-kj4b"
I20240502 01:53:23.685732 31701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:23.711534 31701 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:53:23.712530 31701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:23.712826 31701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:23.713790 31701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:23.715350 31701 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:23.715454 31701 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:23.715574 31701 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:23.715644 31701 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:23.750264 31701 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.130:41639
I20240502 01:53:23.750313 31830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.130:41639 every 8 connection(s)
I20240502 01:53:23.751664 31701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 01:53:23.753665 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 31701
I20240502 01:53:23.753860 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 01:53:23.757120 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.131:0
--local_ip_for_outbound_sockets=127.25.109.131
--webserver_interface=127.25.109.131
--webserver_port=0
--tserver_master_addrs=127.25.109.190:41417
--builtin_ntp_servers=127.25.109.148:40813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:53:23.766602 31831 heartbeater.cc:347] Connected to a master server at 127.25.109.190:41417
I20240502 01:53:23.766949 31831 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:23.767804 31831 heartbeater.cc:511] Master 127.25.109.190:41417 requested a full tablet report, sending...
I20240502 01:53:23.769402 31505 ts_manager.cc:194] Registered new tserver with Master: 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130:41639)
I20240502 01:53:23.770206 31505 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.130:46979
W20240502 01:53:24.083494 31834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:24.083832 31834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:24.084008 31834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:24.091611 31834 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:53:24.091773 31834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:24.091974 31834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.131
I20240502 01:53:24.101245 31834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:40813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.25.109.131
--webserver_port=0
--tserver_master_addrs=127.25.109.190:41417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.131
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:24.102639 31834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:24.104501 31834 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:24.116943 31839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:24.117568 31840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:24.118038 31842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:24.118723 31834 server_base.cc:1034] running on GCE node
I20240502 01:53:24.119388 31834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:24.120450 31834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:24.121665 31834 hybrid_clock.cc:648] HybridClock initialized: now 1714614804121603 us; error 58 us; skew 500 ppm
I20240502 01:53:24.122005 31834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:24.124346 31834 webserver.cc:460] Webserver started at http://127.25.109.131:45279/ using document root <none> and password file <none>
I20240502 01:53:24.125133 31834 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:24.125247 31834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:24.125562 31834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:24.128366 31834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "1864f9ea44c54742b090c594a5be3669"
format_stamp: "Formatted at 2024-05-02 01:53:24 on dist-test-slave-kj4b"
I20240502 01:53:24.129150 31834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "1864f9ea44c54742b090c594a5be3669"
format_stamp: "Formatted at 2024-05-02 01:53:24 on dist-test-slave-kj4b"
I20240502 01:53:24.134507 31834 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240502 01:53:24.138072 31848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:24.139565 31834 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:53:24.139763 31834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "1864f9ea44c54742b090c594a5be3669"
format_stamp: "Formatted at 2024-05-02 01:53:24 on dist-test-slave-kj4b"
I20240502 01:53:24.140059 31834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:24.159255 31834 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:53:24.160342 31834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:24.160640 31834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:24.161612 31834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:24.163081 31834 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:24.163179 31834 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:24.163290 31834 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:24.163357 31834 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:24.193634 31834 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.131:42119
I20240502 01:53:24.193702 31960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.131:42119 every 8 connection(s)
I20240502 01:53:24.195005 31834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 01:53:24.205427 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 31834
I20240502 01:53:24.205654 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 01:53:24.208328 31962 heartbeater.cc:347] Connected to a master server at 127.25.109.190:41417
I20240502 01:53:24.208650 31962 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:24.208854 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.132:0
--local_ip_for_outbound_sockets=127.25.109.132
--webserver_interface=127.25.109.132
--webserver_port=0
--tserver_master_addrs=127.25.109.190:41417
--builtin_ntp_servers=127.25.109.148:40813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:53:24.209465 31962 heartbeater.cc:511] Master 127.25.109.190:41417 requested a full tablet report, sending...
I20240502 01:53:24.210909 31505 ts_manager.cc:194] Registered new tserver with Master: 1864f9ea44c54742b090c594a5be3669 (127.25.109.131:42119)
I20240502 01:53:24.211984 31505 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.131:52365
I20240502 01:53:24.302919 31697 heartbeater.cc:503] Master 127.25.109.190:41417 was elected leader, sending a full tablet report...
W20240502 01:53:24.540051 31965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:24.540366 31965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:24.540498 31965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:24.548283 31965 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:53:24.548427 31965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:24.548581 31965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.132
I20240502 01:53:24.558404 31965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:40813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.25.109.132
--webserver_port=0
--tserver_master_addrs=127.25.109.190:41417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.132
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:24.559782 31965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:24.561661 31965 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:24.573786 31970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:24.574221 31971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:24.574899 31973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:24.575475 31965 server_base.cc:1034] running on GCE node
I20240502 01:53:24.576153 31965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:24.577126 31965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:24.578351 31965 hybrid_clock.cc:648] HybridClock initialized: now 1714614804578296 us; error 43 us; skew 500 ppm
I20240502 01:53:24.578681 31965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:24.581027 31965 webserver.cc:460] Webserver started at http://127.25.109.132:38497/ using document root <none> and password file <none>
I20240502 01:53:24.581895 31965 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:24.582003 31965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:24.582329 31965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:24.585062 31965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "0aea20ce2a1144519ed144383c016d78"
format_stamp: "Formatted at 2024-05-02 01:53:24 on dist-test-slave-kj4b"
I20240502 01:53:24.585819 31965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "0aea20ce2a1144519ed144383c016d78"
format_stamp: "Formatted at 2024-05-02 01:53:24 on dist-test-slave-kj4b"
I20240502 01:53:24.591171 31965 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.002s	sys 0.004s
I20240502 01:53:24.594877 31979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:24.596326 31965 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 01:53:24.596494 31965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "0aea20ce2a1144519ed144383c016d78"
format_stamp: "Formatted at 2024-05-02 01:53:24 on dist-test-slave-kj4b"
I20240502 01:53:24.596690 31965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:24.622200 31965 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:53:24.623072 31965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:24.623338 31965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:24.624322 31965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:24.625865 31965 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:24.625962 31965 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:24.626071 31965 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:24.626137 31965 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:24.656044 31965 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.132:34649
I20240502 01:53:24.656093 32093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.132:34649 every 8 connection(s)
I20240502 01:53:24.657373 31965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 01:53:24.658099 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 31965
I20240502 01:53:24.658295 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 01:53:24.662118 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.133:0
--local_ip_for_outbound_sockets=127.25.109.133
--webserver_interface=127.25.109.133
--webserver_port=0
--tserver_master_addrs=127.25.109.190:41417
--builtin_ntp_servers=127.25.109.148:40813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:53:24.670416 32094 heartbeater.cc:347] Connected to a master server at 127.25.109.190:41417
I20240502 01:53:24.670712 32094 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:24.671451 32094 heartbeater.cc:511] Master 127.25.109.190:41417 requested a full tablet report, sending...
I20240502 01:53:24.672889 31505 ts_manager.cc:194] Registered new tserver with Master: 0aea20ce2a1144519ed144383c016d78 (127.25.109.132:34649)
I20240502 01:53:24.673679 31505 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.132:37537
I20240502 01:53:24.772927 31831 heartbeater.cc:503] Master 127.25.109.190:41417 was elected leader, sending a full tablet report...
W20240502 01:53:24.999370 32097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:24.999693 32097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:24.999831 32097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:25.007441 32097 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:53:25.007592 32097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:25.007762 32097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.133
I20240502 01:53:25.017241 32097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:40813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.25.109.133
--webserver_port=0
--tserver_master_addrs=127.25.109.190:41417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.133
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:25.018600 32097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:25.020426 32097 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:25.034380 32102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:25.034722 32103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:25.035415 32105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:25.035904 32097 server_base.cc:1034] running on GCE node
I20240502 01:53:25.036564 32097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:25.037590 32097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:25.038828 32097 hybrid_clock.cc:648] HybridClock initialized: now 1714614805038754 us; error 74 us; skew 500 ppm
I20240502 01:53:25.039209 32097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:25.041805 32097 webserver.cc:460] Webserver started at http://127.25.109.133:45597/ using document root <none> and password file <none>
I20240502 01:53:25.042604 32097 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:25.042726 32097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:25.043041 32097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:25.045915 32097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "a506969c1a5346c09c19a01f6cbc1a8b"
format_stamp: "Formatted at 2024-05-02 01:53:25 on dist-test-slave-kj4b"
I20240502 01:53:25.046684 32097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "a506969c1a5346c09c19a01f6cbc1a8b"
format_stamp: "Formatted at 2024-05-02 01:53:25 on dist-test-slave-kj4b"
I20240502 01:53:25.052188 32097 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.005s
I20240502 01:53:25.055687 32111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:25.057178 32097 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 01:53:25.057349 32097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "a506969c1a5346c09c19a01f6cbc1a8b"
format_stamp: "Formatted at 2024-05-02 01:53:25 on dist-test-slave-kj4b"
I20240502 01:53:25.057559 32097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:25.106245 32097 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:53:25.107162 32097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:25.107443 32097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:25.108479 32097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:25.110011 32097 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:25.110111 32097 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:25.110220 32097 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:25.110294 32097 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:25.140213 32097 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.133:46379
I20240502 01:53:25.140277 32226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.133:46379 every 8 connection(s)
I20240502 01:53:25.141530 32097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:53:25.151260 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 32097
I20240502 01:53:25.151477 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 01:53:25.154573 32227 heartbeater.cc:347] Connected to a master server at 127.25.109.190:41417
I20240502 01:53:25.154837 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.134:0
--local_ip_for_outbound_sockets=127.25.109.134
--webserver_interface=127.25.109.134
--webserver_port=0
--tserver_master_addrs=127.25.109.190:41417
--builtin_ntp_servers=127.25.109.148:40813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:53:25.154901 32227 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:25.155666 32227 heartbeater.cc:511] Master 127.25.109.190:41417 requested a full tablet report, sending...
I20240502 01:53:25.157083 31505 ts_manager.cc:194] Registered new tserver with Master: a506969c1a5346c09c19a01f6cbc1a8b (127.25.109.133:46379)
I20240502 01:53:25.157871 31505 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.133:39149
I20240502 01:53:25.214560 31962 heartbeater.cc:503] Master 127.25.109.190:41417 was elected leader, sending a full tablet report...
W20240502 01:53:25.479477 32230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:25.479830 32230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:25.479967 32230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:25.487464 32230 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:53:25.487615 32230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:25.487754 32230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.134
I20240502 01:53:25.497047 32230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:40813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.25.109.134
--webserver_port=0
--tserver_master_addrs=127.25.109.190:41417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.134
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:25.498346 32230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:25.500162 32230 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:25.512486 32238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:25.513015 32236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:25.513401 32235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:25.513950 32230 server_base.cc:1034] running on GCE node
I20240502 01:53:25.514526 32230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:25.515523 32230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:25.516724 32230 hybrid_clock.cc:648] HybridClock initialized: now 1714614805516666 us; error 47 us; skew 500 ppm
I20240502 01:53:25.517061 32230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:25.519394 32230 webserver.cc:460] Webserver started at http://127.25.109.134:39535/ using document root <none> and password file <none>
I20240502 01:53:25.520212 32230 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:25.520328 32230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:25.520638 32230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:25.523288 32230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "676092c80dd74040bb49a06db1ac209c"
format_stamp: "Formatted at 2024-05-02 01:53:25 on dist-test-slave-kj4b"
I20240502 01:53:25.524127 32230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "676092c80dd74040bb49a06db1ac209c"
format_stamp: "Formatted at 2024-05-02 01:53:25 on dist-test-slave-kj4b"
I20240502 01:53:25.529376 32230 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.001s	sys 0.003s
I20240502 01:53:25.532930 32244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:25.534453 32230 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20240502 01:53:25.534632 32230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "676092c80dd74040bb49a06db1ac209c"
format_stamp: "Formatted at 2024-05-02 01:53:25 on dist-test-slave-kj4b"
I20240502 01:53:25.534842 32230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:25.567900 32230 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:53:25.568815 32230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:25.569097 32230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:25.570154 32230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:25.571660 32230 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:25.571763 32230 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:25.571899 32230 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:25.571983 32230 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.001s	sys 0.000s
I20240502 01:53:25.602655 32230 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.134:38633
I20240502 01:53:25.602713 32359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.134:38633 every 8 connection(s)
I20240502 01:53:25.603978 32230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 01:53:25.613004 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 32230
I20240502 01:53:25.613183 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 01:53:25.616524 32360 heartbeater.cc:347] Connected to a master server at 127.25.109.190:41417
I20240502 01:53:25.616652 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.135:0
--local_ip_for_outbound_sockets=127.25.109.135
--webserver_interface=127.25.109.135
--webserver_port=0
--tserver_master_addrs=127.25.109.190:41417
--builtin_ntp_servers=127.25.109.148:40813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:53:25.616864 32360 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:25.617677 32360 heartbeater.cc:511] Master 127.25.109.190:41417 requested a full tablet report, sending...
I20240502 01:53:25.619187 31505 ts_manager.cc:194] Registered new tserver with Master: 676092c80dd74040bb49a06db1ac209c (127.25.109.134:38633)
I20240502 01:53:25.620254 31505 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.134:46701
I20240502 01:53:25.676141 32094 heartbeater.cc:503] Master 127.25.109.190:41417 was elected leader, sending a full tablet report...
W20240502 01:53:25.939122 32363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:25.939446 32363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:25.939563 32363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:25.947165 32363 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:53:25.947289 32363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:25.947408 32363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.135
I20240502 01:53:25.956557 32363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:40813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.25.109.135
--webserver_port=0
--tserver_master_addrs=127.25.109.190:41417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.135
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:25.957859 32363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:25.959564 32363 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:25.971639 32368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:25.972174 32369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:25.972630 32371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:25.973201 32363 server_base.cc:1034] running on GCE node
I20240502 01:53:25.973840 32363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:25.974772 32363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:25.975943 32363 hybrid_clock.cc:648] HybridClock initialized: now 1714614805975900 us; error 34 us; skew 500 ppm
I20240502 01:53:25.976255 32363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:25.978412 32363 webserver.cc:460] Webserver started at http://127.25.109.135:39311/ using document root <none> and password file <none>
I20240502 01:53:25.979164 32363 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:25.979276 32363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:25.979575 32363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:25.982174 32363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "facf1046c8054d2d841cf875c63e29e2"
format_stamp: "Formatted at 2024-05-02 01:53:25 on dist-test-slave-kj4b"
I20240502 01:53:25.982901 32363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "facf1046c8054d2d841cf875c63e29e2"
format_stamp: "Formatted at 2024-05-02 01:53:25 on dist-test-slave-kj4b"
I20240502 01:53:25.988019 32363 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.002s
I20240502 01:53:25.991402 32377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:25.992717 32363 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 01:53:25.992877 32363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "facf1046c8054d2d841cf875c63e29e2"
format_stamp: "Formatted at 2024-05-02 01:53:25 on dist-test-slave-kj4b"
I20240502 01:53:25.993068 32363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:26.010811 32363 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:53:26.011644 32363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:26.011946 32363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:26.012871 32363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:26.014379 32363 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:26.014477 32363 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:26.014580 32363 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:26.014649 32363 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:26.045035 32363 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.135:37253
I20240502 01:53:26.045091 32490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.135:37253 every 8 connection(s)
I20240502 01:53:26.046363 32363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 01:53:26.055276 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 32363
I20240502 01:53:26.055476 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 01:53:26.059365 32491 heartbeater.cc:347] Connected to a master server at 127.25.109.190:41417
I20240502 01:53:26.059716 32491 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:26.060514 32491 heartbeater.cc:511] Master 127.25.109.190:41417 requested a full tablet report, sending...
I20240502 01:53:26.061862 31505 ts_manager.cc:194] Registered new tserver with Master: facf1046c8054d2d841cf875c63e29e2 (127.25.109.135:37253)
I20240502 01:53:26.062697 31505 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.135:51691
I20240502 01:53:26.071915 26038 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 01:53:26.088783 31505 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60436:
name: "TestTable"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
owner: "alice"
I20240502 01:53:26.125154 32162 tablet_service.cc:1467] Processing CreateTablet for tablet a865d3891d4e4430a0a0aec06a074cea (DEFAULT_TABLE table=TestTable [id=5e18a6c9d228495e93a8a674b3c7ab8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:26.125154 31896 tablet_service.cc:1467] Processing CreateTablet for tablet a865d3891d4e4430a0a0aec06a074cea (DEFAULT_TABLE table=TestTable [id=5e18a6c9d228495e93a8a674b3c7ab8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:26.126878 31896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a865d3891d4e4430a0a0aec06a074cea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:26.126932 32162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a865d3891d4e4430a0a0aec06a074cea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:26.141603 32508 tablet_bootstrap.cc:492] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b: Bootstrap starting.
I20240502 01:53:26.142594 32509 tablet_bootstrap.cc:492] T a865d3891d4e4430a0a0aec06a074cea P 1864f9ea44c54742b090c594a5be3669: Bootstrap starting.
I20240502 01:53:26.146258 32509 tablet_bootstrap.cc:654] T a865d3891d4e4430a0a0aec06a074cea P 1864f9ea44c54742b090c594a5be3669: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:26.147011 32508 tablet_bootstrap.cc:654] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:26.147459 32509 log.cc:824] T a865d3891d4e4430a0a0aec06a074cea P 1864f9ea44c54742b090c594a5be3669: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:26.148079 32508 log.cc:824] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:26.151525 32509 tablet_bootstrap.cc:492] T a865d3891d4e4430a0a0aec06a074cea P 1864f9ea44c54742b090c594a5be3669: No bootstrap required, opened a new log
I20240502 01:53:26.151855 32509 ts_tablet_manager.cc:1367] T a865d3891d4e4430a0a0aec06a074cea P 1864f9ea44c54742b090c594a5be3669: Time spent bootstrapping tablet: real 0.010s	user 0.004s	sys 0.004s
I20240502 01:53:26.153296 32508 tablet_bootstrap.cc:492] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b: No bootstrap required, opened a new log
I20240502 01:53:26.153556 32508 ts_tablet_manager.cc:1367] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b: Time spent bootstrapping tablet: real 0.012s	user 0.006s	sys 0.000s
I20240502 01:53:26.158283 32509 raft_consensus.cc:348] T a865d3891d4e4430a0a0aec06a074cea P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } }
I20240502 01:53:26.158663 32509 raft_consensus.cc:374] T a865d3891d4e4430a0a0aec06a074cea P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:26.158769 32509 raft_consensus.cc:729] T a865d3891d4e4430a0a0aec06a074cea P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1864f9ea44c54742b090c594a5be3669, State: Initialized, Role: FOLLOWER
I20240502 01:53:26.159466 32509 consensus_queue.cc:260] T a865d3891d4e4430a0a0aec06a074cea P 1864f9ea44c54742b090c594a5be3669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } }
I20240502 01:53:26.159616 32508 raft_consensus.cc:348] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } }
I20240502 01:53:26.160177 32508 raft_consensus.cc:374] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:26.160310 32508 raft_consensus.cc:729] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a506969c1a5346c09c19a01f6cbc1a8b, State: Initialized, Role: FOLLOWER
I20240502 01:53:26.160511 32227 heartbeater.cc:503] Master 127.25.109.190:41417 was elected leader, sending a full tablet report...
I20240502 01:53:26.161115 32508 consensus_queue.cc:260] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } }
I20240502 01:53:26.162297 32509 ts_tablet_manager.cc:1398] T a865d3891d4e4430a0a0aec06a074cea P 1864f9ea44c54742b090c594a5be3669: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.001s
I20240502 01:53:26.163678 32508 ts_tablet_manager.cc:1398] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b: Time spent starting tablet: real 0.010s	user 0.011s	sys 0.000s
W20240502 01:53:26.197999 31963 tablet.cc:2369] T a865d3891d4e4430a0a0aec06a074cea P 1864f9ea44c54742b090c594a5be3669: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:53:26.222407 32513 raft_consensus.cc:482] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:26.222687 32513 raft_consensus.cc:504] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } }
I20240502 01:53:26.224278 32513 leader_election.cc:288] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1864f9ea44c54742b090c594a5be3669 (127.25.109.131:42119)
I20240502 01:53:26.230955 31916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a865d3891d4e4430a0a0aec06a074cea" candidate_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1864f9ea44c54742b090c594a5be3669" is_pre_election: true
I20240502 01:53:26.231478 31916 raft_consensus.cc:2454] T a865d3891d4e4430a0a0aec06a074cea P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a506969c1a5346c09c19a01f6cbc1a8b in term 0.
I20240502 01:53:26.232141 32112 leader_election.cc:302] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1864f9ea44c54742b090c594a5be3669, a506969c1a5346c09c19a01f6cbc1a8b; no voters: 
I20240502 01:53:26.232550 32513 raft_consensus.cc:2781] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:26.232718 32513 raft_consensus.cc:482] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:26.232826 32513 raft_consensus.cc:3037] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:26.234860 32513 raft_consensus.cc:504] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } }
I20240502 01:53:26.235358 32513 leader_election.cc:288] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b [CANDIDATE]: Term 1 election: Requested vote from peers 1864f9ea44c54742b090c594a5be3669 (127.25.109.131:42119)
I20240502 01:53:26.235932 31916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a865d3891d4e4430a0a0aec06a074cea" candidate_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1864f9ea44c54742b090c594a5be3669"
I20240502 01:53:26.236111 31916 raft_consensus.cc:3037] T a865d3891d4e4430a0a0aec06a074cea P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:26.238195 31916 raft_consensus.cc:2454] T a865d3891d4e4430a0a0aec06a074cea P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a506969c1a5346c09c19a01f6cbc1a8b in term 1.
I20240502 01:53:26.238679 32112 leader_election.cc:302] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1864f9ea44c54742b090c594a5be3669, a506969c1a5346c09c19a01f6cbc1a8b; no voters: 
I20240502 01:53:26.239007 32513 raft_consensus.cc:2781] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:26.239478 32513 raft_consensus.cc:686] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b [term 1 LEADER]: Becoming Leader. State: Replica: a506969c1a5346c09c19a01f6cbc1a8b, State: Running, Role: LEADER
I20240502 01:53:26.239977 32513 consensus_queue.cc:237] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } }
I20240502 01:53:26.244211 31504 catalog_manager.cc:5484] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b reported cstate change: term changed from 0 to 1, leader changed from <none> to a506969c1a5346c09c19a01f6cbc1a8b (127.25.109.133). New cstate: current_term: 1 leader_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } }
I20240502 01:53:26.270435 26038 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 01:53:26.272745 26038 ts_itest-base.cc:233] Waiting for 0 tablets on tserver b191086bf5f2432b8d240510817ae5fe to finish bootstrapping
I20240502 01:53:26.283093 26038 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 35f356f20e274db7bf020f78f1bc9ae7 to finish bootstrapping
I20240502 01:53:26.292485 26038 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 1864f9ea44c54742b090c594a5be3669 to finish bootstrapping
I20240502 01:53:26.300441 26038 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 0aea20ce2a1144519ed144383c016d78 to finish bootstrapping
I20240502 01:53:26.309576 26038 ts_itest-base.cc:233] Waiting for 1 tablets on tserver a506969c1a5346c09c19a01f6cbc1a8b to finish bootstrapping
I20240502 01:53:26.316886 26038 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 676092c80dd74040bb49a06db1ac209c to finish bootstrapping
I20240502 01:53:26.325883 26038 ts_itest-base.cc:233] Waiting for 0 tablets on tserver facf1046c8054d2d841cf875c63e29e2 to finish bootstrapping
I20240502 01:53:26.335253 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 31965
I20240502 01:53:26.336328 32089 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 01:53:26.393978 32228 tablet.cc:2369] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:53:26.417596 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 31965
I20240502 01:53:26.431756 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 32097
I20240502 01:53:26.432974 32222 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:26.519955 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 32097
I20240502 01:53:26.535087 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 32230
I20240502 01:53:26.536217 32354 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:26.614748 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 32230
I20240502 01:53:26.629729 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 32363
I20240502 01:53:26.630812 32486 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:26.711019 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 32363
I20240502 01:53:28.393882 32527 raft_consensus.cc:482] T a865d3891d4e4430a0a0aec06a074cea P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:28.394080 32527 raft_consensus.cc:504] T a865d3891d4e4430a0a0aec06a074cea P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } }
I20240502 01:53:28.395424 32527 leader_election.cc:288] T a865d3891d4e4430a0a0aec06a074cea P 1864f9ea44c54742b090c594a5be3669 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a506969c1a5346c09c19a01f6cbc1a8b (127.25.109.133:46379)
W20240502 01:53:28.396551 31852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.133:46379: connect: Connection refused (error 111)
W20240502 01:53:28.401173 31852 leader_election.cc:334] T a865d3891d4e4430a0a0aec06a074cea P 1864f9ea44c54742b090c594a5be3669 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a506969c1a5346c09c19a01f6cbc1a8b (127.25.109.133:46379): Network error: Client connection negotiation failed: client connection to 127.25.109.133:46379: connect: Connection refused (error 111)
I20240502 01:53:28.401379 31852 leader_election.cc:302] T a865d3891d4e4430a0a0aec06a074cea P 1864f9ea44c54742b090c594a5be3669 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 1864f9ea44c54742b090c594a5be3669; no voters: a506969c1a5346c09c19a01f6cbc1a8b
I20240502 01:53:28.401789 32527 raft_consensus.cc:2726] T a865d3891d4e4430a0a0aec06a074cea P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:53:29.915740 32530 raft_consensus.cc:482] T a865d3891d4e4430a0a0aec06a074cea P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:29.915987 32530 raft_consensus.cc:504] T a865d3891d4e4430a0a0aec06a074cea P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } }
I20240502 01:53:29.916661 32530 leader_election.cc:288] T a865d3891d4e4430a0a0aec06a074cea P 1864f9ea44c54742b090c594a5be3669 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a506969c1a5346c09c19a01f6cbc1a8b (127.25.109.133:46379)
W20240502 01:53:29.918766 31852 leader_election.cc:334] T a865d3891d4e4430a0a0aec06a074cea P 1864f9ea44c54742b090c594a5be3669 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a506969c1a5346c09c19a01f6cbc1a8b (127.25.109.133:46379): Network error: Client connection negotiation failed: client connection to 127.25.109.133:46379: connect: Connection refused (error 111)
I20240502 01:53:29.918910 31852 leader_election.cc:302] T a865d3891d4e4430a0a0aec06a074cea P 1864f9ea44c54742b090c594a5be3669 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 1864f9ea44c54742b090c594a5be3669; no voters: a506969c1a5346c09c19a01f6cbc1a8b
I20240502 01:53:29.919167 32530 raft_consensus.cc:2726] T a865d3891d4e4430a0a0aec06a074cea P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:53:30.476081 31504 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60436:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:53:30.500056 31764 tablet_service.cc:1467] Processing CreateTablet for tablet 02b02a474b5e4f6ab173ced2dc21c03e (DEFAULT_TABLE table=rebalance_test_table_0 [id=c997c25d61cd4f25aad452793cf51df8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:30.500087 31762 tablet_service.cc:1467] Processing CreateTablet for tablet fe2daac9229e44c9bee5f7fbaf886728 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c997c25d61cd4f25aad452793cf51df8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:30.502020 31764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02b02a474b5e4f6ab173ced2dc21c03e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:30.500115 31763 tablet_service.cc:1467] Processing CreateTablet for tablet 10aa745ed48d46a3b0558d35dee6a807 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c997c25d61cd4f25aad452793cf51df8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:30.501773 31628 tablet_service.cc:1467] Processing CreateTablet for tablet 10aa745ed48d46a3b0558d35dee6a807 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c997c25d61cd4f25aad452793cf51df8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:30.501672 31627 tablet_service.cc:1467] Processing CreateTablet for tablet fe2daac9229e44c9bee5f7fbaf886728 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c997c25d61cd4f25aad452793cf51df8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:30.502616 31763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10aa745ed48d46a3b0558d35dee6a807. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:30.502698 31629 tablet_service.cc:1467] Processing CreateTablet for tablet 02b02a474b5e4f6ab173ced2dc21c03e (DEFAULT_TABLE table=rebalance_test_table_0 [id=c997c25d61cd4f25aad452793cf51df8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:30.503389 31627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe2daac9229e44c9bee5f7fbaf886728. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:30.503826 31629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02b02a474b5e4f6ab173ced2dc21c03e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:30.504026 31762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe2daac9229e44c9bee5f7fbaf886728. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:30.504122 31628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10aa745ed48d46a3b0558d35dee6a807. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:30.520298 32541 tablet_bootstrap.cc:492] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7: Bootstrap starting.
I20240502 01:53:30.522754 32542 tablet_bootstrap.cc:492] T fe2daac9229e44c9bee5f7fbaf886728 P b191086bf5f2432b8d240510817ae5fe: Bootstrap starting.
I20240502 01:53:30.523270 32541 tablet_bootstrap.cc:654] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:30.524518 32541 log.cc:824] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:30.525965 32542 tablet_bootstrap.cc:654] T fe2daac9229e44c9bee5f7fbaf886728 P b191086bf5f2432b8d240510817ae5fe: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:30.526851 32542 log.cc:824] T fe2daac9229e44c9bee5f7fbaf886728 P b191086bf5f2432b8d240510817ae5fe: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:30.527040 32541 tablet_bootstrap.cc:492] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7: No bootstrap required, opened a new log
I20240502 01:53:30.527266 32541 ts_tablet_manager.cc:1367] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7: Time spent bootstrapping tablet: real 0.007s	user 0.003s	sys 0.002s
I20240502 01:53:30.529100 32542 tablet_bootstrap.cc:492] T fe2daac9229e44c9bee5f7fbaf886728 P b191086bf5f2432b8d240510817ae5fe: No bootstrap required, opened a new log
I20240502 01:53:30.529316 32542 ts_tablet_manager.cc:1367] T fe2daac9229e44c9bee5f7fbaf886728 P b191086bf5f2432b8d240510817ae5fe: Time spent bootstrapping tablet: real 0.007s	user 0.005s	sys 0.000s
I20240502 01:53:30.533597 32541 raft_consensus.cc:348] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } }
I20240502 01:53:30.533926 32541 raft_consensus.cc:374] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:30.534032 32541 raft_consensus.cc:729] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35f356f20e274db7bf020f78f1bc9ae7, State: Initialized, Role: FOLLOWER
I20240502 01:53:30.534632 32541 consensus_queue.cc:260] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } }
I20240502 01:53:30.535323 32542 raft_consensus.cc:348] T fe2daac9229e44c9bee5f7fbaf886728 P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } }
I20240502 01:53:30.535672 32542 raft_consensus.cc:374] T fe2daac9229e44c9bee5f7fbaf886728 P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:30.535777 32542 raft_consensus.cc:729] T fe2daac9229e44c9bee5f7fbaf886728 P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b191086bf5f2432b8d240510817ae5fe, State: Initialized, Role: FOLLOWER
I20240502 01:53:30.536212 32541 ts_tablet_manager.cc:1398] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7: Time spent starting tablet: real 0.009s	user 0.009s	sys 0.001s
I20240502 01:53:30.536578 32541 tablet_bootstrap.cc:492] T fe2daac9229e44c9bee5f7fbaf886728 P 35f356f20e274db7bf020f78f1bc9ae7: Bootstrap starting.
I20240502 01:53:30.536443 32542 consensus_queue.cc:260] T fe2daac9229e44c9bee5f7fbaf886728 P b191086bf5f2432b8d240510817ae5fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } }
I20240502 01:53:30.538017 32542 ts_tablet_manager.cc:1398] T fe2daac9229e44c9bee5f7fbaf886728 P b191086bf5f2432b8d240510817ae5fe: Time spent starting tablet: real 0.009s	user 0.009s	sys 0.000s
I20240502 01:53:30.538264 32541 tablet_bootstrap.cc:654] T fe2daac9229e44c9bee5f7fbaf886728 P 35f356f20e274db7bf020f78f1bc9ae7: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:30.538448 32542 tablet_bootstrap.cc:492] T 02b02a474b5e4f6ab173ced2dc21c03e P b191086bf5f2432b8d240510817ae5fe: Bootstrap starting.
W20240502 01:53:30.538481 31698 tablet.cc:2369] T fe2daac9229e44c9bee5f7fbaf886728 P b191086bf5f2432b8d240510817ae5fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:53:30.540051 32542 tablet_bootstrap.cc:654] T 02b02a474b5e4f6ab173ced2dc21c03e P b191086bf5f2432b8d240510817ae5fe: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:30.542451 32542 tablet_bootstrap.cc:492] T 02b02a474b5e4f6ab173ced2dc21c03e P b191086bf5f2432b8d240510817ae5fe: No bootstrap required, opened a new log
I20240502 01:53:30.542649 32542 ts_tablet_manager.cc:1367] T 02b02a474b5e4f6ab173ced2dc21c03e P b191086bf5f2432b8d240510817ae5fe: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.003s
I20240502 01:53:30.543654 32542 raft_consensus.cc:348] T 02b02a474b5e4f6ab173ced2dc21c03e P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } }
I20240502 01:53:30.543800 32542 raft_consensus.cc:374] T 02b02a474b5e4f6ab173ced2dc21c03e P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:30.543946 32542 raft_consensus.cc:729] T 02b02a474b5e4f6ab173ced2dc21c03e P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b191086bf5f2432b8d240510817ae5fe, State: Initialized, Role: FOLLOWER
I20240502 01:53:30.544193 32542 consensus_queue.cc:260] T 02b02a474b5e4f6ab173ced2dc21c03e P b191086bf5f2432b8d240510817ae5fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } }
I20240502 01:53:30.544641 32542 ts_tablet_manager.cc:1398] T 02b02a474b5e4f6ab173ced2dc21c03e P b191086bf5f2432b8d240510817ae5fe: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 01:53:30.544950 32542 tablet_bootstrap.cc:492] T 10aa745ed48d46a3b0558d35dee6a807 P b191086bf5f2432b8d240510817ae5fe: Bootstrap starting.
I20240502 01:53:30.546351 32542 tablet_bootstrap.cc:654] T 10aa745ed48d46a3b0558d35dee6a807 P b191086bf5f2432b8d240510817ae5fe: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:30.547598 32541 tablet_bootstrap.cc:492] T fe2daac9229e44c9bee5f7fbaf886728 P 35f356f20e274db7bf020f78f1bc9ae7: No bootstrap required, opened a new log
I20240502 01:53:30.547798 32541 ts_tablet_manager.cc:1367] T fe2daac9229e44c9bee5f7fbaf886728 P 35f356f20e274db7bf020f78f1bc9ae7: Time spent bootstrapping tablet: real 0.011s	user 0.005s	sys 0.003s
I20240502 01:53:30.548277 32542 tablet_bootstrap.cc:492] T 10aa745ed48d46a3b0558d35dee6a807 P b191086bf5f2432b8d240510817ae5fe: No bootstrap required, opened a new log
I20240502 01:53:30.548408 32542 ts_tablet_manager.cc:1367] T 10aa745ed48d46a3b0558d35dee6a807 P b191086bf5f2432b8d240510817ae5fe: Time spent bootstrapping tablet: real 0.004s	user 0.002s	sys 0.001s
I20240502 01:53:30.548705 32541 raft_consensus.cc:348] T fe2daac9229e44c9bee5f7fbaf886728 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } }
I20240502 01:53:30.548859 32541 raft_consensus.cc:374] T fe2daac9229e44c9bee5f7fbaf886728 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:30.548940 32541 raft_consensus.cc:729] T fe2daac9229e44c9bee5f7fbaf886728 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35f356f20e274db7bf020f78f1bc9ae7, State: Initialized, Role: FOLLOWER
I20240502 01:53:30.548916 32542 raft_consensus.cc:348] T 10aa745ed48d46a3b0558d35dee6a807 P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } }
I20240502 01:53:30.549115 32542 raft_consensus.cc:374] T 10aa745ed48d46a3b0558d35dee6a807 P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:30.549186 32542 raft_consensus.cc:729] T 10aa745ed48d46a3b0558d35dee6a807 P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b191086bf5f2432b8d240510817ae5fe, State: Initialized, Role: FOLLOWER
I20240502 01:53:30.549167 32541 consensus_queue.cc:260] T fe2daac9229e44c9bee5f7fbaf886728 P 35f356f20e274db7bf020f78f1bc9ae7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } }
I20240502 01:53:30.549376 32542 consensus_queue.cc:260] T 10aa745ed48d46a3b0558d35dee6a807 P b191086bf5f2432b8d240510817ae5fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } }
I20240502 01:53:30.549643 32541 ts_tablet_manager.cc:1398] T fe2daac9229e44c9bee5f7fbaf886728 P 35f356f20e274db7bf020f78f1bc9ae7: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:53:30.549961 32541 tablet_bootstrap.cc:492] T 02b02a474b5e4f6ab173ced2dc21c03e P 35f356f20e274db7bf020f78f1bc9ae7: Bootstrap starting.
I20240502 01:53:30.549988 32542 ts_tablet_manager.cc:1398] T 10aa745ed48d46a3b0558d35dee6a807 P b191086bf5f2432b8d240510817ae5fe: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20240502 01:53:30.552522 32541 tablet_bootstrap.cc:654] T 02b02a474b5e4f6ab173ced2dc21c03e P 35f356f20e274db7bf020f78f1bc9ae7: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:30.554916 32541 tablet_bootstrap.cc:492] T 02b02a474b5e4f6ab173ced2dc21c03e P 35f356f20e274db7bf020f78f1bc9ae7: No bootstrap required, opened a new log
I20240502 01:53:30.555061 32541 ts_tablet_manager.cc:1367] T 02b02a474b5e4f6ab173ced2dc21c03e P 35f356f20e274db7bf020f78f1bc9ae7: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240502 01:53:30.555606 32541 raft_consensus.cc:348] T 02b02a474b5e4f6ab173ced2dc21c03e P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } }
I20240502 01:53:30.555781 32541 raft_consensus.cc:374] T 02b02a474b5e4f6ab173ced2dc21c03e P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:30.555846 32541 raft_consensus.cc:729] T 02b02a474b5e4f6ab173ced2dc21c03e P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35f356f20e274db7bf020f78f1bc9ae7, State: Initialized, Role: FOLLOWER
I20240502 01:53:30.556094 32541 consensus_queue.cc:260] T 02b02a474b5e4f6ab173ced2dc21c03e P 35f356f20e274db7bf020f78f1bc9ae7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } }
I20240502 01:53:30.556581 32541 ts_tablet_manager.cc:1398] T 02b02a474b5e4f6ab173ced2dc21c03e P 35f356f20e274db7bf020f78f1bc9ae7: Time spent starting tablet: real 0.001s	user 0.002s	sys 0.000s
I20240502 01:53:30.598783 32547 raft_consensus.cc:482] T 02b02a474b5e4f6ab173ced2dc21c03e P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:30.599076 32547 raft_consensus.cc:504] T 02b02a474b5e4f6ab173ced2dc21c03e P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } }
I20240502 01:53:30.600996 32547 leader_election.cc:288] T 02b02a474b5e4f6ab173ced2dc21c03e P b191086bf5f2432b8d240510817ae5fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130:41639)
I20240502 01:53:30.607551 31785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02b02a474b5e4f6ab173ced2dc21c03e" candidate_uuid: "b191086bf5f2432b8d240510817ae5fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35f356f20e274db7bf020f78f1bc9ae7" is_pre_election: true
I20240502 01:53:30.608127 31785 raft_consensus.cc:2454] T 02b02a474b5e4f6ab173ced2dc21c03e P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b191086bf5f2432b8d240510817ae5fe in term 0.
I20240502 01:53:30.608814 31582 leader_election.cc:302] T 02b02a474b5e4f6ab173ced2dc21c03e P b191086bf5f2432b8d240510817ae5fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 35f356f20e274db7bf020f78f1bc9ae7, b191086bf5f2432b8d240510817ae5fe; no voters: 
I20240502 01:53:30.609234 32547 raft_consensus.cc:2781] T 02b02a474b5e4f6ab173ced2dc21c03e P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:30.609388 32547 raft_consensus.cc:482] T 02b02a474b5e4f6ab173ced2dc21c03e P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:30.609488 32547 raft_consensus.cc:3037] T 02b02a474b5e4f6ab173ced2dc21c03e P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:30.611567 32547 raft_consensus.cc:504] T 02b02a474b5e4f6ab173ced2dc21c03e P b191086bf5f2432b8d240510817ae5fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } }
I20240502 01:53:30.612159 32547 leader_election.cc:288] T 02b02a474b5e4f6ab173ced2dc21c03e P b191086bf5f2432b8d240510817ae5fe [CANDIDATE]: Term 1 election: Requested vote from peers 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130:41639)
I20240502 01:53:30.612717 31785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02b02a474b5e4f6ab173ced2dc21c03e" candidate_uuid: "b191086bf5f2432b8d240510817ae5fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35f356f20e274db7bf020f78f1bc9ae7"
I20240502 01:53:30.612919 31785 raft_consensus.cc:3037] T 02b02a474b5e4f6ab173ced2dc21c03e P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:30.615067 31785 raft_consensus.cc:2454] T 02b02a474b5e4f6ab173ced2dc21c03e P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b191086bf5f2432b8d240510817ae5fe in term 1.
I20240502 01:53:30.615516 31582 leader_election.cc:302] T 02b02a474b5e4f6ab173ced2dc21c03e P b191086bf5f2432b8d240510817ae5fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 35f356f20e274db7bf020f78f1bc9ae7, b191086bf5f2432b8d240510817ae5fe; no voters: 
I20240502 01:53:30.615891 32547 raft_consensus.cc:2781] T 02b02a474b5e4f6ab173ced2dc21c03e P b191086bf5f2432b8d240510817ae5fe [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:30.616335 32547 raft_consensus.cc:686] T 02b02a474b5e4f6ab173ced2dc21c03e P b191086bf5f2432b8d240510817ae5fe [term 1 LEADER]: Becoming Leader. State: Replica: b191086bf5f2432b8d240510817ae5fe, State: Running, Role: LEADER
I20240502 01:53:30.616873 32547 consensus_queue.cc:237] T 02b02a474b5e4f6ab173ced2dc21c03e P b191086bf5f2432b8d240510817ae5fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } }
I20240502 01:53:30.620771 31505 catalog_manager.cc:5484] T 02b02a474b5e4f6ab173ced2dc21c03e P b191086bf5f2432b8d240510817ae5fe reported cstate change: term changed from 0 to 1, leader changed from <none> to b191086bf5f2432b8d240510817ae5fe (127.25.109.129). New cstate: current_term: 1 leader_uuid: "b191086bf5f2432b8d240510817ae5fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } }
I20240502 01:53:30.747041 32545 raft_consensus.cc:482] T fe2daac9229e44c9bee5f7fbaf886728 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:30.747256 32545 raft_consensus.cc:504] T fe2daac9229e44c9bee5f7fbaf886728 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } }
I20240502 01:53:30.748783 32545 leader_election.cc:288] T fe2daac9229e44c9bee5f7fbaf886728 P 35f356f20e274db7bf020f78f1bc9ae7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b191086bf5f2432b8d240510817ae5fe (127.25.109.129:42243)
I20240502 01:53:30.756184 31649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2daac9229e44c9bee5f7fbaf886728" candidate_uuid: "35f356f20e274db7bf020f78f1bc9ae7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b191086bf5f2432b8d240510817ae5fe" is_pre_election: true
I20240502 01:53:30.756619 31649 raft_consensus.cc:2454] T fe2daac9229e44c9bee5f7fbaf886728 P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35f356f20e274db7bf020f78f1bc9ae7 in term 0.
I20240502 01:53:30.757300 31717 leader_election.cc:302] T fe2daac9229e44c9bee5f7fbaf886728 P 35f356f20e274db7bf020f78f1bc9ae7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 35f356f20e274db7bf020f78f1bc9ae7, b191086bf5f2432b8d240510817ae5fe; no voters: 
I20240502 01:53:30.757733 32545 raft_consensus.cc:2781] T fe2daac9229e44c9bee5f7fbaf886728 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:30.757886 32545 raft_consensus.cc:482] T fe2daac9229e44c9bee5f7fbaf886728 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:30.757987 32545 raft_consensus.cc:3037] T fe2daac9229e44c9bee5f7fbaf886728 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Advancing to term 1
W20240502 01:53:30.758773 31832 tablet.cc:2369] T 02b02a474b5e4f6ab173ced2dc21c03e P 35f356f20e274db7bf020f78f1bc9ae7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:53:30.761274 32545 raft_consensus.cc:504] T fe2daac9229e44c9bee5f7fbaf886728 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } }
I20240502 01:53:30.761941 32545 leader_election.cc:288] T fe2daac9229e44c9bee5f7fbaf886728 P 35f356f20e274db7bf020f78f1bc9ae7 [CANDIDATE]: Term 1 election: Requested vote from peers b191086bf5f2432b8d240510817ae5fe (127.25.109.129:42243)
I20240502 01:53:30.762650 31649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2daac9229e44c9bee5f7fbaf886728" candidate_uuid: "35f356f20e274db7bf020f78f1bc9ae7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b191086bf5f2432b8d240510817ae5fe"
I20240502 01:53:30.762876 31649 raft_consensus.cc:3037] T fe2daac9229e44c9bee5f7fbaf886728 P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:30.765049 31649 raft_consensus.cc:2454] T fe2daac9229e44c9bee5f7fbaf886728 P b191086bf5f2432b8d240510817ae5fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35f356f20e274db7bf020f78f1bc9ae7 in term 1.
I20240502 01:53:30.765585 31717 leader_election.cc:302] T fe2daac9229e44c9bee5f7fbaf886728 P 35f356f20e274db7bf020f78f1bc9ae7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 35f356f20e274db7bf020f78f1bc9ae7, b191086bf5f2432b8d240510817ae5fe; no voters: 
I20240502 01:53:30.765918 32545 raft_consensus.cc:2781] T fe2daac9229e44c9bee5f7fbaf886728 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:30.766356 32545 raft_consensus.cc:686] T fe2daac9229e44c9bee5f7fbaf886728 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 LEADER]: Becoming Leader. State: Replica: 35f356f20e274db7bf020f78f1bc9ae7, State: Running, Role: LEADER
I20240502 01:53:30.766821 32545 consensus_queue.cc:237] T fe2daac9229e44c9bee5f7fbaf886728 P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } }
I20240502 01:53:30.771752 31505 catalog_manager.cc:5484] T fe2daac9229e44c9bee5f7fbaf886728 P 35f356f20e274db7bf020f78f1bc9ae7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130). New cstate: current_term: 1 leader_uuid: "35f356f20e274db7bf020f78f1bc9ae7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } }
I20240502 01:53:30.903848 32547 raft_consensus.cc:482] T 10aa745ed48d46a3b0558d35dee6a807 P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:30.904131 32547 raft_consensus.cc:504] T 10aa745ed48d46a3b0558d35dee6a807 P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } }
I20240502 01:53:30.904579 32545 raft_consensus.cc:482] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:30.904695 32547 leader_election.cc:288] T 10aa745ed48d46a3b0558d35dee6a807 P b191086bf5f2432b8d240510817ae5fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130:41639)
I20240502 01:53:30.904812 32545 raft_consensus.cc:504] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } }
I20240502 01:53:30.905342 32545 leader_election.cc:288] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b191086bf5f2432b8d240510817ae5fe (127.25.109.129:42243)
I20240502 01:53:30.905344 31785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10aa745ed48d46a3b0558d35dee6a807" candidate_uuid: "b191086bf5f2432b8d240510817ae5fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35f356f20e274db7bf020f78f1bc9ae7" is_pre_election: true
I20240502 01:53:30.905615 31785 raft_consensus.cc:2454] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b191086bf5f2432b8d240510817ae5fe in term 0.
I20240502 01:53:30.905882 31649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10aa745ed48d46a3b0558d35dee6a807" candidate_uuid: "35f356f20e274db7bf020f78f1bc9ae7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b191086bf5f2432b8d240510817ae5fe" is_pre_election: true
I20240502 01:53:30.906019 31582 leader_election.cc:302] T 10aa745ed48d46a3b0558d35dee6a807 P b191086bf5f2432b8d240510817ae5fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 35f356f20e274db7bf020f78f1bc9ae7, b191086bf5f2432b8d240510817ae5fe; no voters: 
I20240502 01:53:30.906140 31649 raft_consensus.cc:2454] T 10aa745ed48d46a3b0558d35dee6a807 P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35f356f20e274db7bf020f78f1bc9ae7 in term 0.
I20240502 01:53:30.906332 32547 raft_consensus.cc:2781] T 10aa745ed48d46a3b0558d35dee6a807 P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:30.906446 32547 raft_consensus.cc:482] T 10aa745ed48d46a3b0558d35dee6a807 P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:30.906555 32547 raft_consensus.cc:3037] T 10aa745ed48d46a3b0558d35dee6a807 P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:30.906551 31717 leader_election.cc:302] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 35f356f20e274db7bf020f78f1bc9ae7, b191086bf5f2432b8d240510817ae5fe; no voters: 
I20240502 01:53:30.906850 32545 raft_consensus.cc:2781] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:30.906947 32545 raft_consensus.cc:482] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:30.907050 32545 raft_consensus.cc:3037] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:30.908542 32547 raft_consensus.cc:504] T 10aa745ed48d46a3b0558d35dee6a807 P b191086bf5f2432b8d240510817ae5fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } }
I20240502 01:53:30.909054 32547 leader_election.cc:288] T 10aa745ed48d46a3b0558d35dee6a807 P b191086bf5f2432b8d240510817ae5fe [CANDIDATE]: Term 1 election: Requested vote from peers 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130:41639)
I20240502 01:53:30.908972 32545 raft_consensus.cc:504] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } }
I20240502 01:53:30.909533 32545 leader_election.cc:288] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [CANDIDATE]: Term 1 election: Requested vote from peers b191086bf5f2432b8d240510817ae5fe (127.25.109.129:42243)
I20240502 01:53:30.909582 31785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10aa745ed48d46a3b0558d35dee6a807" candidate_uuid: "b191086bf5f2432b8d240510817ae5fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35f356f20e274db7bf020f78f1bc9ae7"
I20240502 01:53:30.909907 31785 raft_consensus.cc:2379] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b191086bf5f2432b8d240510817ae5fe in current term 1: Already voted for candidate 35f356f20e274db7bf020f78f1bc9ae7 in this term.
I20240502 01:53:30.909951 31649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10aa745ed48d46a3b0558d35dee6a807" candidate_uuid: "35f356f20e274db7bf020f78f1bc9ae7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b191086bf5f2432b8d240510817ae5fe"
I20240502 01:53:30.910210 31649 raft_consensus.cc:2379] T 10aa745ed48d46a3b0558d35dee6a807 P b191086bf5f2432b8d240510817ae5fe [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 35f356f20e274db7bf020f78f1bc9ae7 in current term 1: Already voted for candidate b191086bf5f2432b8d240510817ae5fe in this term.
I20240502 01:53:30.910605 31582 leader_election.cc:302] T 10aa745ed48d46a3b0558d35dee6a807 P b191086bf5f2432b8d240510817ae5fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: b191086bf5f2432b8d240510817ae5fe; no voters: 35f356f20e274db7bf020f78f1bc9ae7
I20240502 01:53:30.910882 32547 raft_consensus.cc:2726] T 10aa745ed48d46a3b0558d35dee6a807 P b191086bf5f2432b8d240510817ae5fe [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 01:53:30.910892 31717 leader_election.cc:302] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 35f356f20e274db7bf020f78f1bc9ae7; no voters: b191086bf5f2432b8d240510817ae5fe
I20240502 01:53:30.911136 32545 raft_consensus.cc:2726] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 01:53:31.148129 32547 consensus_queue.cc:1035] T 02b02a474b5e4f6ab173ced2dc21c03e P b191086bf5f2432b8d240510817ae5fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:31.349016 32545 consensus_queue.cc:1035] T fe2daac9229e44c9bee5f7fbaf886728 P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:31.473590 32564 raft_consensus.cc:482] T a865d3891d4e4430a0a0aec06a074cea P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:31.473817 32564 raft_consensus.cc:504] T a865d3891d4e4430a0a0aec06a074cea P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } }
I20240502 01:53:31.474440 32564 leader_election.cc:288] T a865d3891d4e4430a0a0aec06a074cea P 1864f9ea44c54742b090c594a5be3669 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a506969c1a5346c09c19a01f6cbc1a8b (127.25.109.133:46379)
W20240502 01:53:31.476459 31852 leader_election.cc:334] T a865d3891d4e4430a0a0aec06a074cea P 1864f9ea44c54742b090c594a5be3669 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a506969c1a5346c09c19a01f6cbc1a8b (127.25.109.133:46379): Network error: Client connection negotiation failed: client connection to 127.25.109.133:46379: connect: Connection refused (error 111)
I20240502 01:53:31.476593 31852 leader_election.cc:302] T a865d3891d4e4430a0a0aec06a074cea P 1864f9ea44c54742b090c594a5be3669 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 1864f9ea44c54742b090c594a5be3669; no voters: a506969c1a5346c09c19a01f6cbc1a8b
I20240502 01:53:31.476866 32564 raft_consensus.cc:2726] T a865d3891d4e4430a0a0aec06a074cea P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:53:32.795778 32569 raft_consensus.cc:482] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:32.796005 32569 raft_consensus.cc:504] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } }
I20240502 01:53:32.796594 32569 leader_election.cc:288] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b191086bf5f2432b8d240510817ae5fe (127.25.109.129:42243)
I20240502 01:53:32.797223 31649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10aa745ed48d46a3b0558d35dee6a807" candidate_uuid: "35f356f20e274db7bf020f78f1bc9ae7" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b191086bf5f2432b8d240510817ae5fe" is_pre_election: true
I20240502 01:53:32.797492 31649 raft_consensus.cc:2454] T 10aa745ed48d46a3b0558d35dee6a807 P b191086bf5f2432b8d240510817ae5fe [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35f356f20e274db7bf020f78f1bc9ae7 in term 1.
I20240502 01:53:32.797979 31717 leader_election.cc:302] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 35f356f20e274db7bf020f78f1bc9ae7, b191086bf5f2432b8d240510817ae5fe; no voters: 
I20240502 01:53:32.798257 32569 raft_consensus.cc:2781] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:53:32.798372 32569 raft_consensus.cc:482] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:32.798454 32569 raft_consensus.cc:3037] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:32.800379 32569 raft_consensus.cc:504] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } }
I20240502 01:53:32.800884 32569 leader_election.cc:288] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [CANDIDATE]: Term 2 election: Requested vote from peers b191086bf5f2432b8d240510817ae5fe (127.25.109.129:42243)
I20240502 01:53:32.801481 31649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10aa745ed48d46a3b0558d35dee6a807" candidate_uuid: "35f356f20e274db7bf020f78f1bc9ae7" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b191086bf5f2432b8d240510817ae5fe"
I20240502 01:53:32.801702 31649 raft_consensus.cc:3037] T 10aa745ed48d46a3b0558d35dee6a807 P b191086bf5f2432b8d240510817ae5fe [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:32.803555 31649 raft_consensus.cc:2454] T 10aa745ed48d46a3b0558d35dee6a807 P b191086bf5f2432b8d240510817ae5fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35f356f20e274db7bf020f78f1bc9ae7 in term 2.
I20240502 01:53:32.804054 31717 leader_election.cc:302] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 35f356f20e274db7bf020f78f1bc9ae7, b191086bf5f2432b8d240510817ae5fe; no voters: 
I20240502 01:53:32.804354 32569 raft_consensus.cc:2781] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:53:32.804742 32569 raft_consensus.cc:686] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [term 2 LEADER]: Becoming Leader. State: Replica: 35f356f20e274db7bf020f78f1bc9ae7, State: Running, Role: LEADER
I20240502 01:53:32.805042 32569 consensus_queue.cc:237] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } }
I20240502 01:53:32.807397 31505 catalog_manager.cc:5484] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 reported cstate change: term changed from 0 to 2, leader changed from <none> to 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130). New cstate: current_term: 2 leader_uuid: "35f356f20e274db7bf020f78f1bc9ae7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } }
I20240502 01:53:33.182612 31649 raft_consensus.cc:1261] T 10aa745ed48d46a3b0558d35dee6a807 P b191086bf5f2432b8d240510817ae5fe [term 2 FOLLOWER]: Refusing update from remote peer 35f356f20e274db7bf020f78f1bc9ae7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240502 01:53:33.183372 32569 consensus_queue.cc:1035] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:33.461480 32576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:33.461846 32576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:33.469542 32576 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'

INSERT report
    rows total: 2000
    time total: 179.068 ms
  time per row: 0.0895339 ms
I20240502 01:53:33.795008 31503 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60436:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:53:33.804961 31627 tablet_service.cc:1467] Processing CreateTablet for tablet f2a3a225d7f343d9b0ac75e41f0d2a11 (DEFAULT_TABLE table=rebalance_test_table_1 [id=76d2ee30f2bb480087daf717aa077ba2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:33.804983 31896 tablet_service.cc:1467] Processing CreateTablet for tablet f2a3a225d7f343d9b0ac75e41f0d2a11 (DEFAULT_TABLE table=rebalance_test_table_1 [id=76d2ee30f2bb480087daf717aa077ba2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:33.805325 31895 tablet_service.cc:1467] Processing CreateTablet for tablet 238e56f8d83748e8b2cf1100e92db4c6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=76d2ee30f2bb480087daf717aa077ba2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:33.805481 31896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2a3a225d7f343d9b0ac75e41f0d2a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:33.805476 31627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2a3a225d7f343d9b0ac75e41f0d2a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:33.805987 31894 tablet_service.cc:1467] Processing CreateTablet for tablet 559610821917470fbcf5641d36acf409 (DEFAULT_TABLE table=rebalance_test_table_1 [id=76d2ee30f2bb480087daf717aa077ba2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:33.806416 31894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 559610821917470fbcf5641d36acf409. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:33.806799 31764 tablet_service.cc:1467] Processing CreateTablet for tablet 238e56f8d83748e8b2cf1100e92db4c6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=76d2ee30f2bb480087daf717aa077ba2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:33.807222 31764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 238e56f8d83748e8b2cf1100e92db4c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:33.808339 31628 tablet_service.cc:1467] Processing CreateTablet for tablet 559610821917470fbcf5641d36acf409 (DEFAULT_TABLE table=rebalance_test_table_1 [id=76d2ee30f2bb480087daf717aa077ba2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:33.808722 31628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 559610821917470fbcf5641d36acf409. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:33.812465 31895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 238e56f8d83748e8b2cf1100e92db4c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:33.817520 32634 tablet_bootstrap.cc:492] T f2a3a225d7f343d9b0ac75e41f0d2a11 P 1864f9ea44c54742b090c594a5be3669: Bootstrap starting.
I20240502 01:53:33.819532 32634 tablet_bootstrap.cc:654] T f2a3a225d7f343d9b0ac75e41f0d2a11 P 1864f9ea44c54742b090c594a5be3669: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:33.821718 32635 tablet_bootstrap.cc:492] T f2a3a225d7f343d9b0ac75e41f0d2a11 P b191086bf5f2432b8d240510817ae5fe: Bootstrap starting.
I20240502 01:53:33.823634 32634 tablet_bootstrap.cc:492] T f2a3a225d7f343d9b0ac75e41f0d2a11 P 1864f9ea44c54742b090c594a5be3669: No bootstrap required, opened a new log
I20240502 01:53:33.823844 32634 ts_tablet_manager.cc:1367] T f2a3a225d7f343d9b0ac75e41f0d2a11 P 1864f9ea44c54742b090c594a5be3669: Time spent bootstrapping tablet: real 0.006s	user 0.006s	sys 0.000s
I20240502 01:53:33.823940 32635 tablet_bootstrap.cc:654] T f2a3a225d7f343d9b0ac75e41f0d2a11 P b191086bf5f2432b8d240510817ae5fe: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:33.824904 32634 raft_consensus.cc:348] T f2a3a225d7f343d9b0ac75e41f0d2a11 P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } }
I20240502 01:53:33.825240 32634 raft_consensus.cc:374] T f2a3a225d7f343d9b0ac75e41f0d2a11 P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:33.825397 32634 raft_consensus.cc:729] T f2a3a225d7f343d9b0ac75e41f0d2a11 P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1864f9ea44c54742b090c594a5be3669, State: Initialized, Role: FOLLOWER
I20240502 01:53:33.825733 32638 tablet_bootstrap.cc:492] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7: Bootstrap starting.
I20240502 01:53:33.825708 32634 consensus_queue.cc:260] T f2a3a225d7f343d9b0ac75e41f0d2a11 P 1864f9ea44c54742b090c594a5be3669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } }
I20240502 01:53:33.826489 32635 tablet_bootstrap.cc:492] T f2a3a225d7f343d9b0ac75e41f0d2a11 P b191086bf5f2432b8d240510817ae5fe: No bootstrap required, opened a new log
I20240502 01:53:33.826524 32634 ts_tablet_manager.cc:1398] T f2a3a225d7f343d9b0ac75e41f0d2a11 P 1864f9ea44c54742b090c594a5be3669: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 01:53:33.826699 32635 ts_tablet_manager.cc:1367] T f2a3a225d7f343d9b0ac75e41f0d2a11 P b191086bf5f2432b8d240510817ae5fe: Time spent bootstrapping tablet: real 0.005s	user 0.000s	sys 0.004s
I20240502 01:53:33.826908 32634 tablet_bootstrap.cc:492] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669: Bootstrap starting.
I20240502 01:53:33.827517 32635 raft_consensus.cc:348] T f2a3a225d7f343d9b0ac75e41f0d2a11 P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } }
I20240502 01:53:33.827917 32635 raft_consensus.cc:374] T f2a3a225d7f343d9b0ac75e41f0d2a11 P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:33.828068 32635 raft_consensus.cc:729] T f2a3a225d7f343d9b0ac75e41f0d2a11 P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b191086bf5f2432b8d240510817ae5fe, State: Initialized, Role: FOLLOWER
I20240502 01:53:33.828299 32638 tablet_bootstrap.cc:654] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:33.828510 32634 tablet_bootstrap.cc:654] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:33.828405 32635 consensus_queue.cc:260] T f2a3a225d7f343d9b0ac75e41f0d2a11 P b191086bf5f2432b8d240510817ae5fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } }
I20240502 01:53:33.828955 32635 ts_tablet_manager.cc:1398] T f2a3a225d7f343d9b0ac75e41f0d2a11 P b191086bf5f2432b8d240510817ae5fe: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 01:53:33.829277 32635 tablet_bootstrap.cc:492] T 559610821917470fbcf5641d36acf409 P b191086bf5f2432b8d240510817ae5fe: Bootstrap starting.
I20240502 01:53:33.831073 32635 tablet_bootstrap.cc:654] T 559610821917470fbcf5641d36acf409 P b191086bf5f2432b8d240510817ae5fe: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:33.836371 32635 tablet_bootstrap.cc:492] T 559610821917470fbcf5641d36acf409 P b191086bf5f2432b8d240510817ae5fe: No bootstrap required, opened a new log
I20240502 01:53:33.836570 32635 ts_tablet_manager.cc:1367] T 559610821917470fbcf5641d36acf409 P b191086bf5f2432b8d240510817ae5fe: Time spent bootstrapping tablet: real 0.007s	user 0.006s	sys 0.000s
I20240502 01:53:33.836652 32634 tablet_bootstrap.cc:492] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669: No bootstrap required, opened a new log
I20240502 01:53:33.836608 32638 tablet_bootstrap.cc:492] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7: No bootstrap required, opened a new log
I20240502 01:53:33.836820 32634 ts_tablet_manager.cc:1367] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669: Time spent bootstrapping tablet: real 0.010s	user 0.002s	sys 0.006s
I20240502 01:53:33.836859 32638 ts_tablet_manager.cc:1367] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7: Time spent bootstrapping tablet: real 0.011s	user 0.005s	sys 0.005s
I20240502 01:53:33.837415 32635 raft_consensus.cc:348] T 559610821917470fbcf5641d36acf409 P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } }
I20240502 01:53:33.837652 32635 raft_consensus.cc:374] T 559610821917470fbcf5641d36acf409 P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:33.837597 32634 raft_consensus.cc:348] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } }
I20240502 01:53:33.837742 32635 raft_consensus.cc:729] T 559610821917470fbcf5641d36acf409 P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b191086bf5f2432b8d240510817ae5fe, State: Initialized, Role: FOLLOWER
I20240502 01:53:33.837811 32634 raft_consensus.cc:374] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:33.837903 32634 raft_consensus.cc:729] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1864f9ea44c54742b090c594a5be3669, State: Initialized, Role: FOLLOWER
I20240502 01:53:33.837843 32638 raft_consensus.cc:348] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } }
I20240502 01:53:33.837982 32635 consensus_queue.cc:260] T 559610821917470fbcf5641d36acf409 P b191086bf5f2432b8d240510817ae5fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } }
I20240502 01:53:33.838135 32634 consensus_queue.cc:260] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } }
I20240502 01:53:33.838232 32638 raft_consensus.cc:374] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:33.838348 32638 raft_consensus.cc:729] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35f356f20e274db7bf020f78f1bc9ae7, State: Initialized, Role: FOLLOWER
I20240502 01:53:33.838604 32635 ts_tablet_manager.cc:1398] T 559610821917470fbcf5641d36acf409 P b191086bf5f2432b8d240510817ae5fe: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:53:33.838744 32634 ts_tablet_manager.cc:1398] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240502 01:53:33.838644 32638 consensus_queue.cc:260] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } }
I20240502 01:53:33.839027 32634 tablet_bootstrap.cc:492] T 238e56f8d83748e8b2cf1100e92db4c6 P 1864f9ea44c54742b090c594a5be3669: Bootstrap starting.
I20240502 01:53:33.839890 32638 ts_tablet_manager.cc:1398] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:53:33.840823 32634 tablet_bootstrap.cc:654] T 238e56f8d83748e8b2cf1100e92db4c6 P 1864f9ea44c54742b090c594a5be3669: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:33.843565 32634 tablet_bootstrap.cc:492] T 238e56f8d83748e8b2cf1100e92db4c6 P 1864f9ea44c54742b090c594a5be3669: No bootstrap required, opened a new log
I20240502 01:53:33.843717 32634 ts_tablet_manager.cc:1367] T 238e56f8d83748e8b2cf1100e92db4c6 P 1864f9ea44c54742b090c594a5be3669: Time spent bootstrapping tablet: real 0.005s	user 0.000s	sys 0.003s
I20240502 01:53:33.844416 32634 raft_consensus.cc:348] T 238e56f8d83748e8b2cf1100e92db4c6 P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } }
I20240502 01:53:33.844559 32634 raft_consensus.cc:374] T 238e56f8d83748e8b2cf1100e92db4c6 P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:33.844622 32634 raft_consensus.cc:729] T 238e56f8d83748e8b2cf1100e92db4c6 P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1864f9ea44c54742b090c594a5be3669, State: Initialized, Role: FOLLOWER
I20240502 01:53:33.844805 32634 consensus_queue.cc:260] T 238e56f8d83748e8b2cf1100e92db4c6 P 1864f9ea44c54742b090c594a5be3669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } }
I20240502 01:53:33.845229 32634 ts_tablet_manager.cc:1398] T 238e56f8d83748e8b2cf1100e92db4c6 P 1864f9ea44c54742b090c594a5be3669: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 01:53:33.863492 32639 raft_consensus.cc:482] T f2a3a225d7f343d9b0ac75e41f0d2a11 P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:33.863675 32639 raft_consensus.cc:504] T f2a3a225d7f343d9b0ac75e41f0d2a11 P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } }
I20240502 01:53:33.864470 32639 leader_election.cc:288] T f2a3a225d7f343d9b0ac75e41f0d2a11 P 1864f9ea44c54742b090c594a5be3669 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b191086bf5f2432b8d240510817ae5fe (127.25.109.129:42243)
I20240502 01:53:33.870549 31648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2a3a225d7f343d9b0ac75e41f0d2a11" candidate_uuid: "1864f9ea44c54742b090c594a5be3669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b191086bf5f2432b8d240510817ae5fe" is_pre_election: true
I20240502 01:53:33.870790 31648 raft_consensus.cc:2454] T f2a3a225d7f343d9b0ac75e41f0d2a11 P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1864f9ea44c54742b090c594a5be3669 in term 0.
I20240502 01:53:33.871270 31850 leader_election.cc:302] T f2a3a225d7f343d9b0ac75e41f0d2a11 P 1864f9ea44c54742b090c594a5be3669 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1864f9ea44c54742b090c594a5be3669, b191086bf5f2432b8d240510817ae5fe; no voters: 
I20240502 01:53:33.871584 32639 raft_consensus.cc:2781] T f2a3a225d7f343d9b0ac75e41f0d2a11 P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:33.871714 32639 raft_consensus.cc:482] T f2a3a225d7f343d9b0ac75e41f0d2a11 P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:33.871815 32639 raft_consensus.cc:3037] T f2a3a225d7f343d9b0ac75e41f0d2a11 P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:33.874118 32639 raft_consensus.cc:504] T f2a3a225d7f343d9b0ac75e41f0d2a11 P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } }
I20240502 01:53:33.874671 32639 leader_election.cc:288] T f2a3a225d7f343d9b0ac75e41f0d2a11 P 1864f9ea44c54742b090c594a5be3669 [CANDIDATE]: Term 1 election: Requested vote from peers b191086bf5f2432b8d240510817ae5fe (127.25.109.129:42243)
I20240502 01:53:33.875303 31648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2a3a225d7f343d9b0ac75e41f0d2a11" candidate_uuid: "1864f9ea44c54742b090c594a5be3669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b191086bf5f2432b8d240510817ae5fe"
I20240502 01:53:33.875528 31648 raft_consensus.cc:3037] T f2a3a225d7f343d9b0ac75e41f0d2a11 P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:33.877593 31648 raft_consensus.cc:2454] T f2a3a225d7f343d9b0ac75e41f0d2a11 P b191086bf5f2432b8d240510817ae5fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1864f9ea44c54742b090c594a5be3669 in term 1.
I20240502 01:53:33.878065 31850 leader_election.cc:302] T f2a3a225d7f343d9b0ac75e41f0d2a11 P 1864f9ea44c54742b090c594a5be3669 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1864f9ea44c54742b090c594a5be3669, b191086bf5f2432b8d240510817ae5fe; no voters: 
I20240502 01:53:33.878391 32639 raft_consensus.cc:2781] T f2a3a225d7f343d9b0ac75e41f0d2a11 P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:33.878885 32639 raft_consensus.cc:686] T f2a3a225d7f343d9b0ac75e41f0d2a11 P 1864f9ea44c54742b090c594a5be3669 [term 1 LEADER]: Becoming Leader. State: Replica: 1864f9ea44c54742b090c594a5be3669, State: Running, Role: LEADER
I20240502 01:53:33.879412 32639 consensus_queue.cc:237] T f2a3a225d7f343d9b0ac75e41f0d2a11 P 1864f9ea44c54742b090c594a5be3669 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } }
I20240502 01:53:33.883913 31503 catalog_manager.cc:5484] T f2a3a225d7f343d9b0ac75e41f0d2a11 P 1864f9ea44c54742b090c594a5be3669 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1864f9ea44c54742b090c594a5be3669 (127.25.109.131). New cstate: current_term: 1 leader_uuid: "1864f9ea44c54742b090c594a5be3669" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } }
I20240502 01:53:33.957979 32639 raft_consensus.cc:482] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:33.958236 32639 raft_consensus.cc:504] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } }
I20240502 01:53:33.958776 32639 leader_election.cc:288] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b191086bf5f2432b8d240510817ae5fe (127.25.109.129:42243)
I20240502 01:53:33.959472 31648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "559610821917470fbcf5641d36acf409" candidate_uuid: "1864f9ea44c54742b090c594a5be3669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b191086bf5f2432b8d240510817ae5fe" is_pre_election: true
I20240502 01:53:33.959806 31648 raft_consensus.cc:2454] T 559610821917470fbcf5641d36acf409 P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1864f9ea44c54742b090c594a5be3669 in term 0.
I20240502 01:53:33.960279 31850 leader_election.cc:302] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1864f9ea44c54742b090c594a5be3669, b191086bf5f2432b8d240510817ae5fe; no voters: 
I20240502 01:53:33.960565 32639 raft_consensus.cc:2781] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:33.960659 32639 raft_consensus.cc:482] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:33.960758 32639 raft_consensus.cc:3037] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:33.962679 32639 raft_consensus.cc:504] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } }
I20240502 01:53:33.963212 32639 leader_election.cc:288] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669 [CANDIDATE]: Term 1 election: Requested vote from peers b191086bf5f2432b8d240510817ae5fe (127.25.109.129:42243)
I20240502 01:53:33.963737 31648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "559610821917470fbcf5641d36acf409" candidate_uuid: "1864f9ea44c54742b090c594a5be3669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b191086bf5f2432b8d240510817ae5fe"
I20240502 01:53:33.963999 31648 raft_consensus.cc:3037] T 559610821917470fbcf5641d36acf409 P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:33.964159 32621 raft_consensus.cc:482] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:33.964322 32621 raft_consensus.cc:504] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } }
I20240502 01:53:33.965063 32621 leader_election.cc:288] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1864f9ea44c54742b090c594a5be3669 (127.25.109.131:42119)
I20240502 01:53:33.966372 31648 raft_consensus.cc:2454] T 559610821917470fbcf5641d36acf409 P b191086bf5f2432b8d240510817ae5fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1864f9ea44c54742b090c594a5be3669 in term 1.
I20240502 01:53:33.966848 31850 leader_election.cc:302] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1864f9ea44c54742b090c594a5be3669, b191086bf5f2432b8d240510817ae5fe; no voters: 
I20240502 01:53:33.967180 32639 raft_consensus.cc:2781] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:33.967352 32639 raft_consensus.cc:686] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669 [term 1 LEADER]: Becoming Leader. State: Replica: 1864f9ea44c54742b090c594a5be3669, State: Running, Role: LEADER
I20240502 01:53:33.967636 32639 consensus_queue.cc:237] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } }
I20240502 01:53:33.970681 31503 catalog_manager.cc:5484] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1864f9ea44c54742b090c594a5be3669 (127.25.109.131). New cstate: current_term: 1 leader_uuid: "1864f9ea44c54742b090c594a5be3669" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } }
I20240502 01:53:33.973070 31916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "238e56f8d83748e8b2cf1100e92db4c6" candidate_uuid: "35f356f20e274db7bf020f78f1bc9ae7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1864f9ea44c54742b090c594a5be3669" is_pre_election: true
I20240502 01:53:33.973317 31916 raft_consensus.cc:2454] T 238e56f8d83748e8b2cf1100e92db4c6 P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35f356f20e274db7bf020f78f1bc9ae7 in term 0.
I20240502 01:53:33.973779 31716 leader_election.cc:302] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1864f9ea44c54742b090c594a5be3669, 35f356f20e274db7bf020f78f1bc9ae7; no voters: 
I20240502 01:53:33.974166 32621 raft_consensus.cc:2781] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:33.974265 32621 raft_consensus.cc:482] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:33.974336 32621 raft_consensus.cc:3037] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:33.976500 32621 raft_consensus.cc:504] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } }
I20240502 01:53:33.977041 32621 leader_election.cc:288] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7 [CANDIDATE]: Term 1 election: Requested vote from peers 1864f9ea44c54742b090c594a5be3669 (127.25.109.131:42119)
I20240502 01:53:33.977569 31916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "238e56f8d83748e8b2cf1100e92db4c6" candidate_uuid: "35f356f20e274db7bf020f78f1bc9ae7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1864f9ea44c54742b090c594a5be3669"
I20240502 01:53:33.977777 31916 raft_consensus.cc:3037] T 238e56f8d83748e8b2cf1100e92db4c6 P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:33.979749 31916 raft_consensus.cc:2454] T 238e56f8d83748e8b2cf1100e92db4c6 P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35f356f20e274db7bf020f78f1bc9ae7 in term 1.
I20240502 01:53:33.980206 31716 leader_election.cc:302] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1864f9ea44c54742b090c594a5be3669, 35f356f20e274db7bf020f78f1bc9ae7; no voters: 
I20240502 01:53:33.980549 32621 raft_consensus.cc:2781] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:33.980690 32621 raft_consensus.cc:686] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 LEADER]: Becoming Leader. State: Replica: 35f356f20e274db7bf020f78f1bc9ae7, State: Running, Role: LEADER
I20240502 01:53:33.980966 32621 consensus_queue.cc:237] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } }
I20240502 01:53:33.983497 31503 catalog_manager.cc:5484] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130). New cstate: current_term: 1 leader_uuid: "35f356f20e274db7bf020f78f1bc9ae7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } }
I20240502 01:53:34.330719 32639 consensus_queue.cc:1035] T f2a3a225d7f343d9b0ac75e41f0d2a11 P 1864f9ea44c54742b090c594a5be3669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:34.359194 32651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:34.359560 32651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:34.367230 32651 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 01:53:34.481829 32645 consensus_queue.cc:1035] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:34.510263 31916 raft_consensus.cc:1261] T 238e56f8d83748e8b2cf1100e92db4c6 P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Refusing update from remote peer 35f356f20e274db7bf020f78f1bc9ae7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:34.511344 32621 consensus_queue.cc:1035] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:34.544922 32687 mvcc.cc:205] Tried to move back new op lower bound from 7023062280222765056 to 7023062278067978240. Current Snapshot: MvccSnapshot[applied={T|T < 7023062280222765056}]

INSERT report
    rows total: 2000
    time total: 199.245 ms
  time per row: 0.0996226 ms
I20240502 01:53:34.708571 31498 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60436:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:53:34.717667 31764 tablet_service.cc:1467] Processing CreateTablet for tablet 65576af2c4bb4f25b4d14963cbe65ca7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=7340ad22b7574a6d81f7eaa977a06414]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:34.717963 31895 tablet_service.cc:1467] Processing CreateTablet for tablet 65576af2c4bb4f25b4d14963cbe65ca7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=7340ad22b7574a6d81f7eaa977a06414]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:34.718252 31764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65576af2c4bb4f25b4d14963cbe65ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:34.718417 31895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65576af2c4bb4f25b4d14963cbe65ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:34.718817 31896 tablet_service.cc:1467] Processing CreateTablet for tablet 704700e8872b4675b41d6c172731a5e0 (DEFAULT_TABLE table=rebalance_test_table_2 [id=7340ad22b7574a6d81f7eaa977a06414]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:34.719159 31896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 704700e8872b4675b41d6c172731a5e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:34.719329 31628 tablet_service.cc:1467] Processing CreateTablet for tablet 86f2e43af7fb4e02bb853f0748d62ddb (DEFAULT_TABLE table=rebalance_test_table_2 [id=7340ad22b7574a6d81f7eaa977a06414]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:34.719746 31628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86f2e43af7fb4e02bb853f0748d62ddb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:34.725013 31763 tablet_service.cc:1467] Processing CreateTablet for tablet 704700e8872b4675b41d6c172731a5e0 (DEFAULT_TABLE table=rebalance_test_table_2 [id=7340ad22b7574a6d81f7eaa977a06414]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:34.725565 31763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 704700e8872b4675b41d6c172731a5e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:34.726966 31762 tablet_service.cc:1467] Processing CreateTablet for tablet 86f2e43af7fb4e02bb853f0748d62ddb (DEFAULT_TABLE table=rebalance_test_table_2 [id=7340ad22b7574a6d81f7eaa977a06414]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:34.727386 31762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86f2e43af7fb4e02bb853f0748d62ddb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:34.734565 32704 tablet_bootstrap.cc:492] T 86f2e43af7fb4e02bb853f0748d62ddb P b191086bf5f2432b8d240510817ae5fe: Bootstrap starting.
I20240502 01:53:34.735673 32705 tablet_bootstrap.cc:492] T 86f2e43af7fb4e02bb853f0748d62ddb P 35f356f20e274db7bf020f78f1bc9ae7: Bootstrap starting.
I20240502 01:53:34.737419 32705 tablet_bootstrap.cc:654] T 86f2e43af7fb4e02bb853f0748d62ddb P 35f356f20e274db7bf020f78f1bc9ae7: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:34.737433 32704 tablet_bootstrap.cc:654] T 86f2e43af7fb4e02bb853f0748d62ddb P b191086bf5f2432b8d240510817ae5fe: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:34.746960 32704 tablet_bootstrap.cc:492] T 86f2e43af7fb4e02bb853f0748d62ddb P b191086bf5f2432b8d240510817ae5fe: No bootstrap required, opened a new log
I20240502 01:53:34.747079 32705 tablet_bootstrap.cc:492] T 86f2e43af7fb4e02bb853f0748d62ddb P 35f356f20e274db7bf020f78f1bc9ae7: No bootstrap required, opened a new log
I20240502 01:53:34.747061 32703 tablet_bootstrap.cc:492] T 704700e8872b4675b41d6c172731a5e0 P 1864f9ea44c54742b090c594a5be3669: Bootstrap starting.
I20240502 01:53:34.747155 32704 ts_tablet_manager.cc:1367] T 86f2e43af7fb4e02bb853f0748d62ddb P b191086bf5f2432b8d240510817ae5fe: Time spent bootstrapping tablet: real 0.013s	user 0.008s	sys 0.000s
I20240502 01:53:34.747232 32705 ts_tablet_manager.cc:1367] T 86f2e43af7fb4e02bb853f0748d62ddb P 35f356f20e274db7bf020f78f1bc9ae7: Time spent bootstrapping tablet: real 0.012s	user 0.006s	sys 0.003s
I20240502 01:53:34.748050 32704 raft_consensus.cc:348] T 86f2e43af7fb4e02bb853f0748d62ddb P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } }
I20240502 01:53:34.748041 32705 raft_consensus.cc:348] T 86f2e43af7fb4e02bb853f0748d62ddb P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } }
I20240502 01:53:34.748270 32705 raft_consensus.cc:374] T 86f2e43af7fb4e02bb853f0748d62ddb P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:34.748270 32704 raft_consensus.cc:374] T 86f2e43af7fb4e02bb853f0748d62ddb P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:34.748364 32704 raft_consensus.cc:729] T 86f2e43af7fb4e02bb853f0748d62ddb P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b191086bf5f2432b8d240510817ae5fe, State: Initialized, Role: FOLLOWER
I20240502 01:53:34.748373 32705 raft_consensus.cc:729] T 86f2e43af7fb4e02bb853f0748d62ddb P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35f356f20e274db7bf020f78f1bc9ae7, State: Initialized, Role: FOLLOWER
I20240502 01:53:34.748622 32705 consensus_queue.cc:260] T 86f2e43af7fb4e02bb853f0748d62ddb P 35f356f20e274db7bf020f78f1bc9ae7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } }
I20240502 01:53:34.748607 32704 consensus_queue.cc:260] T 86f2e43af7fb4e02bb853f0748d62ddb P b191086bf5f2432b8d240510817ae5fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } }
I20240502 01:53:34.749315 32705 ts_tablet_manager.cc:1398] T 86f2e43af7fb4e02bb853f0748d62ddb P 35f356f20e274db7bf020f78f1bc9ae7: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:53:34.749626 32705 tablet_bootstrap.cc:492] T 65576af2c4bb4f25b4d14963cbe65ca7 P 35f356f20e274db7bf020f78f1bc9ae7: Bootstrap starting.
I20240502 01:53:34.751298 32705 tablet_bootstrap.cc:654] T 65576af2c4bb4f25b4d14963cbe65ca7 P 35f356f20e274db7bf020f78f1bc9ae7: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:34.754528 32703 tablet_bootstrap.cc:654] T 704700e8872b4675b41d6c172731a5e0 P 1864f9ea44c54742b090c594a5be3669: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:34.758527 32704 ts_tablet_manager.cc:1398] T 86f2e43af7fb4e02bb853f0748d62ddb P b191086bf5f2432b8d240510817ae5fe: Time spent starting tablet: real 0.011s	user 0.002s	sys 0.000s
I20240502 01:53:34.760267 32705 tablet_bootstrap.cc:492] T 65576af2c4bb4f25b4d14963cbe65ca7 P 35f356f20e274db7bf020f78f1bc9ae7: No bootstrap required, opened a new log
I20240502 01:53:34.760362 32703 tablet_bootstrap.cc:492] T 704700e8872b4675b41d6c172731a5e0 P 1864f9ea44c54742b090c594a5be3669: No bootstrap required, opened a new log
I20240502 01:53:34.760483 32705 ts_tablet_manager.cc:1367] T 65576af2c4bb4f25b4d14963cbe65ca7 P 35f356f20e274db7bf020f78f1bc9ae7: Time spent bootstrapping tablet: real 0.011s	user 0.002s	sys 0.007s
I20240502 01:53:34.760535 32703 ts_tablet_manager.cc:1367] T 704700e8872b4675b41d6c172731a5e0 P 1864f9ea44c54742b090c594a5be3669: Time spent bootstrapping tablet: real 0.014s	user 0.000s	sys 0.005s
I20240502 01:53:34.761358 32703 raft_consensus.cc:348] T 704700e8872b4675b41d6c172731a5e0 P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } }
I20240502 01:53:34.761389 32705 raft_consensus.cc:348] T 65576af2c4bb4f25b4d14963cbe65ca7 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } }
I20240502 01:53:34.761583 32703 raft_consensus.cc:374] T 704700e8872b4675b41d6c172731a5e0 P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:34.761615 32705 raft_consensus.cc:374] T 65576af2c4bb4f25b4d14963cbe65ca7 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:34.761663 32703 raft_consensus.cc:729] T 704700e8872b4675b41d6c172731a5e0 P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1864f9ea44c54742b090c594a5be3669, State: Initialized, Role: FOLLOWER
I20240502 01:53:34.761694 32705 raft_consensus.cc:729] T 65576af2c4bb4f25b4d14963cbe65ca7 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35f356f20e274db7bf020f78f1bc9ae7, State: Initialized, Role: FOLLOWER
I20240502 01:53:34.761960 32705 consensus_queue.cc:260] T 65576af2c4bb4f25b4d14963cbe65ca7 P 35f356f20e274db7bf020f78f1bc9ae7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } }
I20240502 01:53:34.761925 32703 consensus_queue.cc:260] T 704700e8872b4675b41d6c172731a5e0 P 1864f9ea44c54742b090c594a5be3669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } }
I20240502 01:53:34.762454 32705 ts_tablet_manager.cc:1398] T 65576af2c4bb4f25b4d14963cbe65ca7 P 35f356f20e274db7bf020f78f1bc9ae7: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:53:34.762774 32703 ts_tablet_manager.cc:1398] T 704700e8872b4675b41d6c172731a5e0 P 1864f9ea44c54742b090c594a5be3669: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:53:34.763113 32705 tablet_bootstrap.cc:492] T 704700e8872b4675b41d6c172731a5e0 P 35f356f20e274db7bf020f78f1bc9ae7: Bootstrap starting.
I20240502 01:53:34.763231 32703 tablet_bootstrap.cc:492] T 65576af2c4bb4f25b4d14963cbe65ca7 P 1864f9ea44c54742b090c594a5be3669: Bootstrap starting.
I20240502 01:53:34.765112 32705 tablet_bootstrap.cc:654] T 704700e8872b4675b41d6c172731a5e0 P 35f356f20e274db7bf020f78f1bc9ae7: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:34.765146 32703 tablet_bootstrap.cc:654] T 65576af2c4bb4f25b4d14963cbe65ca7 P 1864f9ea44c54742b090c594a5be3669: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:34.768292 32703 tablet_bootstrap.cc:492] T 65576af2c4bb4f25b4d14963cbe65ca7 P 1864f9ea44c54742b090c594a5be3669: No bootstrap required, opened a new log
I20240502 01:53:34.768478 32703 ts_tablet_manager.cc:1367] T 65576af2c4bb4f25b4d14963cbe65ca7 P 1864f9ea44c54742b090c594a5be3669: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240502 01:53:34.769306 32703 raft_consensus.cc:348] T 65576af2c4bb4f25b4d14963cbe65ca7 P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } }
I20240502 01:53:34.769542 32703 raft_consensus.cc:374] T 65576af2c4bb4f25b4d14963cbe65ca7 P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:34.769624 32703 raft_consensus.cc:729] T 65576af2c4bb4f25b4d14963cbe65ca7 P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1864f9ea44c54742b090c594a5be3669, State: Initialized, Role: FOLLOWER
I20240502 01:53:34.769975 32705 tablet_bootstrap.cc:492] T 704700e8872b4675b41d6c172731a5e0 P 35f356f20e274db7bf020f78f1bc9ae7: No bootstrap required, opened a new log
I20240502 01:53:34.769914 32703 consensus_queue.cc:260] T 65576af2c4bb4f25b4d14963cbe65ca7 P 1864f9ea44c54742b090c594a5be3669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } }
I20240502 01:53:34.770277 32705 ts_tablet_manager.cc:1367] T 704700e8872b4675b41d6c172731a5e0 P 35f356f20e274db7bf020f78f1bc9ae7: Time spent bootstrapping tablet: real 0.007s	user 0.005s	sys 0.001s
I20240502 01:53:34.771242 32703 ts_tablet_manager.cc:1398] T 65576af2c4bb4f25b4d14963cbe65ca7 P 1864f9ea44c54742b090c594a5be3669: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:53:34.771577 32705 raft_consensus.cc:348] T 704700e8872b4675b41d6c172731a5e0 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } }
I20240502 01:53:34.771886 32705 raft_consensus.cc:374] T 704700e8872b4675b41d6c172731a5e0 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:34.772044 32705 raft_consensus.cc:729] T 704700e8872b4675b41d6c172731a5e0 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35f356f20e274db7bf020f78f1bc9ae7, State: Initialized, Role: FOLLOWER
I20240502 01:53:34.772361 32705 consensus_queue.cc:260] T 704700e8872b4675b41d6c172731a5e0 P 35f356f20e274db7bf020f78f1bc9ae7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } }
I20240502 01:53:34.772959 32705 ts_tablet_manager.cc:1398] T 704700e8872b4675b41d6c172731a5e0 P 35f356f20e274db7bf020f78f1bc9ae7: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:53:34.829701 32568 raft_consensus.cc:482] T 86f2e43af7fb4e02bb853f0748d62ddb P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:34.829922 32568 raft_consensus.cc:504] T 86f2e43af7fb4e02bb853f0748d62ddb P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } }
I20240502 01:53:34.830510 32568 leader_election.cc:288] T 86f2e43af7fb4e02bb853f0748d62ddb P b191086bf5f2432b8d240510817ae5fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130:41639)
I20240502 01:53:34.831166 31785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86f2e43af7fb4e02bb853f0748d62ddb" candidate_uuid: "b191086bf5f2432b8d240510817ae5fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35f356f20e274db7bf020f78f1bc9ae7" is_pre_election: true
I20240502 01:53:34.831419 31785 raft_consensus.cc:2454] T 86f2e43af7fb4e02bb853f0748d62ddb P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b191086bf5f2432b8d240510817ae5fe in term 0.
I20240502 01:53:34.831915 31582 leader_election.cc:302] T 86f2e43af7fb4e02bb853f0748d62ddb P b191086bf5f2432b8d240510817ae5fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 35f356f20e274db7bf020f78f1bc9ae7, b191086bf5f2432b8d240510817ae5fe; no voters: 
I20240502 01:53:34.832207 32568 raft_consensus.cc:2781] T 86f2e43af7fb4e02bb853f0748d62ddb P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:34.832321 32568 raft_consensus.cc:482] T 86f2e43af7fb4e02bb853f0748d62ddb P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:34.832406 32568 raft_consensus.cc:3037] T 86f2e43af7fb4e02bb853f0748d62ddb P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:34.834328 32568 raft_consensus.cc:504] T 86f2e43af7fb4e02bb853f0748d62ddb P b191086bf5f2432b8d240510817ae5fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } }
I20240502 01:53:34.834805 32568 leader_election.cc:288] T 86f2e43af7fb4e02bb853f0748d62ddb P b191086bf5f2432b8d240510817ae5fe [CANDIDATE]: Term 1 election: Requested vote from peers 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130:41639)
I20240502 01:53:34.835369 31785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86f2e43af7fb4e02bb853f0748d62ddb" candidate_uuid: "b191086bf5f2432b8d240510817ae5fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35f356f20e274db7bf020f78f1bc9ae7"
I20240502 01:53:34.835577 31785 raft_consensus.cc:3037] T 86f2e43af7fb4e02bb853f0748d62ddb P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:34.837502 31785 raft_consensus.cc:2454] T 86f2e43af7fb4e02bb853f0748d62ddb P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b191086bf5f2432b8d240510817ae5fe in term 1.
I20240502 01:53:34.837929 31582 leader_election.cc:302] T 86f2e43af7fb4e02bb853f0748d62ddb P b191086bf5f2432b8d240510817ae5fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 35f356f20e274db7bf020f78f1bc9ae7, b191086bf5f2432b8d240510817ae5fe; no voters: 
I20240502 01:53:34.838241 32568 raft_consensus.cc:2781] T 86f2e43af7fb4e02bb853f0748d62ddb P b191086bf5f2432b8d240510817ae5fe [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:34.838585 32568 raft_consensus.cc:686] T 86f2e43af7fb4e02bb853f0748d62ddb P b191086bf5f2432b8d240510817ae5fe [term 1 LEADER]: Becoming Leader. State: Replica: b191086bf5f2432b8d240510817ae5fe, State: Running, Role: LEADER
I20240502 01:53:34.838878 32568 consensus_queue.cc:237] T 86f2e43af7fb4e02bb853f0748d62ddb P b191086bf5f2432b8d240510817ae5fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } }
I20240502 01:53:34.841495 31498 catalog_manager.cc:5484] T 86f2e43af7fb4e02bb853f0748d62ddb P b191086bf5f2432b8d240510817ae5fe reported cstate change: term changed from 0 to 1, leader changed from <none> to b191086bf5f2432b8d240510817ae5fe (127.25.109.129). New cstate: current_term: 1 leader_uuid: "b191086bf5f2432b8d240510817ae5fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } }
I20240502 01:53:34.852934 32689 raft_consensus.cc:482] T 704700e8872b4675b41d6c172731a5e0 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:34.853137 32689 raft_consensus.cc:504] T 704700e8872b4675b41d6c172731a5e0 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } }
I20240502 01:53:34.853753 32689 leader_election.cc:288] T 704700e8872b4675b41d6c172731a5e0 P 35f356f20e274db7bf020f78f1bc9ae7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1864f9ea44c54742b090c594a5be3669 (127.25.109.131:42119)
I20240502 01:53:34.854382 31916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704700e8872b4675b41d6c172731a5e0" candidate_uuid: "35f356f20e274db7bf020f78f1bc9ae7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1864f9ea44c54742b090c594a5be3669" is_pre_election: true
I20240502 01:53:34.854598 31916 raft_consensus.cc:2454] T 704700e8872b4675b41d6c172731a5e0 P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35f356f20e274db7bf020f78f1bc9ae7 in term 0.
I20240502 01:53:34.854964 31716 leader_election.cc:302] T 704700e8872b4675b41d6c172731a5e0 P 35f356f20e274db7bf020f78f1bc9ae7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1864f9ea44c54742b090c594a5be3669, 35f356f20e274db7bf020f78f1bc9ae7; no voters: 
I20240502 01:53:34.855221 32689 raft_consensus.cc:2781] T 704700e8872b4675b41d6c172731a5e0 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:34.855355 32689 raft_consensus.cc:482] T 704700e8872b4675b41d6c172731a5e0 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:34.855424 32689 raft_consensus.cc:3037] T 704700e8872b4675b41d6c172731a5e0 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:34.857374 32689 raft_consensus.cc:504] T 704700e8872b4675b41d6c172731a5e0 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } }
I20240502 01:53:34.857813 32689 leader_election.cc:288] T 704700e8872b4675b41d6c172731a5e0 P 35f356f20e274db7bf020f78f1bc9ae7 [CANDIDATE]: Term 1 election: Requested vote from peers 1864f9ea44c54742b090c594a5be3669 (127.25.109.131:42119)
I20240502 01:53:34.858450 31916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "704700e8872b4675b41d6c172731a5e0" candidate_uuid: "35f356f20e274db7bf020f78f1bc9ae7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1864f9ea44c54742b090c594a5be3669"
I20240502 01:53:34.858666 31916 raft_consensus.cc:3037] T 704700e8872b4675b41d6c172731a5e0 P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:34.860749 31916 raft_consensus.cc:2454] T 704700e8872b4675b41d6c172731a5e0 P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35f356f20e274db7bf020f78f1bc9ae7 in term 1.
I20240502 01:53:34.861188 31716 leader_election.cc:302] T 704700e8872b4675b41d6c172731a5e0 P 35f356f20e274db7bf020f78f1bc9ae7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1864f9ea44c54742b090c594a5be3669, 35f356f20e274db7bf020f78f1bc9ae7; no voters: 
I20240502 01:53:34.861536 32689 raft_consensus.cc:2781] T 704700e8872b4675b41d6c172731a5e0 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:34.861668 32689 raft_consensus.cc:686] T 704700e8872b4675b41d6c172731a5e0 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 LEADER]: Becoming Leader. State: Replica: 35f356f20e274db7bf020f78f1bc9ae7, State: Running, Role: LEADER
I20240502 01:53:34.861951 32689 consensus_queue.cc:237] T 704700e8872b4675b41d6c172731a5e0 P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } }
I20240502 01:53:34.864564 31498 catalog_manager.cc:5484] T 704700e8872b4675b41d6c172731a5e0 P 35f356f20e274db7bf020f78f1bc9ae7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130). New cstate: current_term: 1 leader_uuid: "35f356f20e274db7bf020f78f1bc9ae7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } }
I20240502 01:53:34.936720 32639 raft_consensus.cc:482] T a865d3891d4e4430a0a0aec06a074cea P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:34.936935 32639 raft_consensus.cc:504] T a865d3891d4e4430a0a0aec06a074cea P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } }
I20240502 01:53:34.937453 32639 leader_election.cc:288] T a865d3891d4e4430a0a0aec06a074cea P 1864f9ea44c54742b090c594a5be3669 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a506969c1a5346c09c19a01f6cbc1a8b (127.25.109.133:46379)
W20240502 01:53:34.938527 31852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.133:46379: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20240502 01:53:34.940016 31852 leader_election.cc:334] T a865d3891d4e4430a0a0aec06a074cea P 1864f9ea44c54742b090c594a5be3669 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a506969c1a5346c09c19a01f6cbc1a8b (127.25.109.133:46379): Network error: Client connection negotiation failed: client connection to 127.25.109.133:46379: connect: Connection refused (error 111)
I20240502 01:53:34.940128 31852 leader_election.cc:302] T a865d3891d4e4430a0a0aec06a074cea P 1864f9ea44c54742b090c594a5be3669 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 1864f9ea44c54742b090c594a5be3669; no voters: a506969c1a5346c09c19a01f6cbc1a8b
I20240502 01:53:34.940443 32639 raft_consensus.cc:2726] T a865d3891d4e4430a0a0aec06a074cea P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:53:34.963419 32689 raft_consensus.cc:482] T 65576af2c4bb4f25b4d14963cbe65ca7 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:34.963640 32689 raft_consensus.cc:504] T 65576af2c4bb4f25b4d14963cbe65ca7 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } }
I20240502 01:53:34.964215 32689 leader_election.cc:288] T 65576af2c4bb4f25b4d14963cbe65ca7 P 35f356f20e274db7bf020f78f1bc9ae7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1864f9ea44c54742b090c594a5be3669 (127.25.109.131:42119)
I20240502 01:53:34.964902 31916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65576af2c4bb4f25b4d14963cbe65ca7" candidate_uuid: "35f356f20e274db7bf020f78f1bc9ae7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1864f9ea44c54742b090c594a5be3669" is_pre_election: true
I20240502 01:53:34.965133 31916 raft_consensus.cc:2454] T 65576af2c4bb4f25b4d14963cbe65ca7 P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35f356f20e274db7bf020f78f1bc9ae7 in term 0.
I20240502 01:53:34.965574 31716 leader_election.cc:302] T 65576af2c4bb4f25b4d14963cbe65ca7 P 35f356f20e274db7bf020f78f1bc9ae7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1864f9ea44c54742b090c594a5be3669, 35f356f20e274db7bf020f78f1bc9ae7; no voters: 
I20240502 01:53:34.965902 32689 raft_consensus.cc:2781] T 65576af2c4bb4f25b4d14963cbe65ca7 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:34.965994 32689 raft_consensus.cc:482] T 65576af2c4bb4f25b4d14963cbe65ca7 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:34.966096 32689 raft_consensus.cc:3037] T 65576af2c4bb4f25b4d14963cbe65ca7 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:34.968070 32689 raft_consensus.cc:504] T 65576af2c4bb4f25b4d14963cbe65ca7 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } }
I20240502 01:53:34.968580 32689 leader_election.cc:288] T 65576af2c4bb4f25b4d14963cbe65ca7 P 35f356f20e274db7bf020f78f1bc9ae7 [CANDIDATE]: Term 1 election: Requested vote from peers 1864f9ea44c54742b090c594a5be3669 (127.25.109.131:42119)
I20240502 01:53:34.969166 31916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65576af2c4bb4f25b4d14963cbe65ca7" candidate_uuid: "35f356f20e274db7bf020f78f1bc9ae7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1864f9ea44c54742b090c594a5be3669"
I20240502 01:53:34.969369 31916 raft_consensus.cc:3037] T 65576af2c4bb4f25b4d14963cbe65ca7 P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:34.971278 31916 raft_consensus.cc:2454] T 65576af2c4bb4f25b4d14963cbe65ca7 P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35f356f20e274db7bf020f78f1bc9ae7 in term 1.
I20240502 01:53:34.971729 31716 leader_election.cc:302] T 65576af2c4bb4f25b4d14963cbe65ca7 P 35f356f20e274db7bf020f78f1bc9ae7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1864f9ea44c54742b090c594a5be3669, 35f356f20e274db7bf020f78f1bc9ae7; no voters: 
I20240502 01:53:34.972082 32689 raft_consensus.cc:2781] T 65576af2c4bb4f25b4d14963cbe65ca7 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:34.972263 32689 raft_consensus.cc:686] T 65576af2c4bb4f25b4d14963cbe65ca7 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 LEADER]: Becoming Leader. State: Replica: 35f356f20e274db7bf020f78f1bc9ae7, State: Running, Role: LEADER
I20240502 01:53:34.972530 32689 consensus_queue.cc:237] T 65576af2c4bb4f25b4d14963cbe65ca7 P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } }
I20240502 01:53:34.975158 31498 catalog_manager.cc:5484] T 65576af2c4bb4f25b4d14963cbe65ca7 P 35f356f20e274db7bf020f78f1bc9ae7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130). New cstate: current_term: 1 leader_uuid: "35f356f20e274db7bf020f78f1bc9ae7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } }
I20240502 01:53:35.269248 32689 consensus_queue.cc:1035] T 704700e8872b4675b41d6c172731a5e0 P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:35.314013 32568 consensus_queue.cc:1035] T 86f2e43af7fb4e02bb853f0748d62ddb P b191086bf5f2432b8d240510817ae5fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:35.372232 32718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:35.372573 32718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:35.380226 32718 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 01:53:35.448413 32689 consensus_queue.cc:1035] T 65576af2c4bb4f25b4d14963cbe65ca7 P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 164.032 ms
  time per row: 0.0820159 ms
I20240502 01:53:35.684870 31505 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60436:
name: "rebalance_test_table_3"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:53:35.694108 31628 tablet_service.cc:1467] Processing CreateTablet for tablet 1313391ae5b2440d93445deda2642a14 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ae2e57387f5a496cb85e41e055010f5b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:35.694316 31896 tablet_service.cc:1467] Processing CreateTablet for tablet 1313391ae5b2440d93445deda2642a14 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ae2e57387f5a496cb85e41e055010f5b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:35.694665 31628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1313391ae5b2440d93445deda2642a14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:35.694613 31627 tablet_service.cc:1467] Processing CreateTablet for tablet 5575decad4be448e835db06b570c69a8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ae2e57387f5a496cb85e41e055010f5b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:35.694784 31896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1313391ae5b2440d93445deda2642a14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:35.695008 31627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5575decad4be448e835db06b570c69a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:35.695762 31895 tablet_service.cc:1467] Processing CreateTablet for tablet 5575decad4be448e835db06b570c69a8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ae2e57387f5a496cb85e41e055010f5b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:35.696209 31895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5575decad4be448e835db06b570c69a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:35.702389   307 tablet_bootstrap.cc:492] T 1313391ae5b2440d93445deda2642a14 P b191086bf5f2432b8d240510817ae5fe: Bootstrap starting.
I20240502 01:53:35.703631 31628 tablet_service.cc:1467] Processing CreateTablet for tablet 6b56adc64eba4898b80687a2679b70c8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ae2e57387f5a496cb85e41e055010f5b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:35.703845 31763 tablet_service.cc:1467] Processing CreateTablet for tablet 6b56adc64eba4898b80687a2679b70c8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ae2e57387f5a496cb85e41e055010f5b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:35.704114 31628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b56adc64eba4898b80687a2679b70c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:35.704296 31763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b56adc64eba4898b80687a2679b70c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:35.704913   307 tablet_bootstrap.cc:654] T 1313391ae5b2440d93445deda2642a14 P b191086bf5f2432b8d240510817ae5fe: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:35.712244   309 tablet_bootstrap.cc:492] T 6b56adc64eba4898b80687a2679b70c8 P 35f356f20e274db7bf020f78f1bc9ae7: Bootstrap starting.
I20240502 01:53:35.712968   306 tablet_bootstrap.cc:492] T 1313391ae5b2440d93445deda2642a14 P 1864f9ea44c54742b090c594a5be3669: Bootstrap starting.
I20240502 01:53:35.714210   307 tablet_bootstrap.cc:492] T 1313391ae5b2440d93445deda2642a14 P b191086bf5f2432b8d240510817ae5fe: No bootstrap required, opened a new log
I20240502 01:53:35.714373   307 ts_tablet_manager.cc:1367] T 1313391ae5b2440d93445deda2642a14 P b191086bf5f2432b8d240510817ae5fe: Time spent bootstrapping tablet: real 0.012s	user 0.002s	sys 0.008s
I20240502 01:53:35.715176   307 raft_consensus.cc:348] T 1313391ae5b2440d93445deda2642a14 P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } }
I20240502 01:53:35.715358   307 raft_consensus.cc:374] T 1313391ae5b2440d93445deda2642a14 P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:35.715404   306 tablet_bootstrap.cc:654] T 1313391ae5b2440d93445deda2642a14 P 1864f9ea44c54742b090c594a5be3669: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:35.715449   307 raft_consensus.cc:729] T 1313391ae5b2440d93445deda2642a14 P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b191086bf5f2432b8d240510817ae5fe, State: Initialized, Role: FOLLOWER
I20240502 01:53:35.715682   307 consensus_queue.cc:260] T 1313391ae5b2440d93445deda2642a14 P b191086bf5f2432b8d240510817ae5fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } }
I20240502 01:53:35.715834   309 tablet_bootstrap.cc:654] T 6b56adc64eba4898b80687a2679b70c8 P 35f356f20e274db7bf020f78f1bc9ae7: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:35.716429   307 ts_tablet_manager.cc:1398] T 1313391ae5b2440d93445deda2642a14 P b191086bf5f2432b8d240510817ae5fe: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.003s
I20240502 01:53:35.716728   307 tablet_bootstrap.cc:492] T 5575decad4be448e835db06b570c69a8 P b191086bf5f2432b8d240510817ae5fe: Bootstrap starting.
I20240502 01:53:35.718317   307 tablet_bootstrap.cc:654] T 5575decad4be448e835db06b570c69a8 P b191086bf5f2432b8d240510817ae5fe: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:35.723824   307 tablet_bootstrap.cc:492] T 5575decad4be448e835db06b570c69a8 P b191086bf5f2432b8d240510817ae5fe: No bootstrap required, opened a new log
I20240502 01:53:35.724035   307 ts_tablet_manager.cc:1367] T 5575decad4be448e835db06b570c69a8 P b191086bf5f2432b8d240510817ae5fe: Time spent bootstrapping tablet: real 0.007s	user 0.001s	sys 0.006s
I20240502 01:53:35.724047   306 tablet_bootstrap.cc:492] T 1313391ae5b2440d93445deda2642a14 P 1864f9ea44c54742b090c594a5be3669: No bootstrap required, opened a new log
I20240502 01:53:35.724217   306 ts_tablet_manager.cc:1367] T 1313391ae5b2440d93445deda2642a14 P 1864f9ea44c54742b090c594a5be3669: Time spent bootstrapping tablet: real 0.011s	user 0.010s	sys 0.000s
I20240502 01:53:35.724800   307 raft_consensus.cc:348] T 5575decad4be448e835db06b570c69a8 P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } }
I20240502 01:53:35.725028   307 raft_consensus.cc:374] T 5575decad4be448e835db06b570c69a8 P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:35.725087   307 raft_consensus.cc:729] T 5575decad4be448e835db06b570c69a8 P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b191086bf5f2432b8d240510817ae5fe, State: Initialized, Role: FOLLOWER
I20240502 01:53:35.725065   306 raft_consensus.cc:348] T 1313391ae5b2440d93445deda2642a14 P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } }
I20240502 01:53:35.725291   306 raft_consensus.cc:374] T 1313391ae5b2440d93445deda2642a14 P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:35.725375   306 raft_consensus.cc:729] T 1313391ae5b2440d93445deda2642a14 P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1864f9ea44c54742b090c594a5be3669, State: Initialized, Role: FOLLOWER
I20240502 01:53:35.725353   307 consensus_queue.cc:260] T 5575decad4be448e835db06b570c69a8 P b191086bf5f2432b8d240510817ae5fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } }
I20240502 01:53:35.725634   306 consensus_queue.cc:260] T 1313391ae5b2440d93445deda2642a14 P 1864f9ea44c54742b090c594a5be3669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } }
I20240502 01:53:35.725965   307 ts_tablet_manager.cc:1398] T 5575decad4be448e835db06b570c69a8 P b191086bf5f2432b8d240510817ae5fe: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240502 01:53:35.726217   307 tablet_bootstrap.cc:492] T 6b56adc64eba4898b80687a2679b70c8 P b191086bf5f2432b8d240510817ae5fe: Bootstrap starting.
I20240502 01:53:35.726277   306 ts_tablet_manager.cc:1398] T 1313391ae5b2440d93445deda2642a14 P 1864f9ea44c54742b090c594a5be3669: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:53:35.726536   306 tablet_bootstrap.cc:492] T 5575decad4be448e835db06b570c69a8 P 1864f9ea44c54742b090c594a5be3669: Bootstrap starting.
I20240502 01:53:35.727793   307 tablet_bootstrap.cc:654] T 6b56adc64eba4898b80687a2679b70c8 P b191086bf5f2432b8d240510817ae5fe: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:35.728979   306 tablet_bootstrap.cc:654] T 5575decad4be448e835db06b570c69a8 P 1864f9ea44c54742b090c594a5be3669: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:35.730983   307 tablet_bootstrap.cc:492] T 6b56adc64eba4898b80687a2679b70c8 P b191086bf5f2432b8d240510817ae5fe: No bootstrap required, opened a new log
I20240502 01:53:35.731143   307 ts_tablet_manager.cc:1367] T 6b56adc64eba4898b80687a2679b70c8 P b191086bf5f2432b8d240510817ae5fe: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240502 01:53:35.731961   307 raft_consensus.cc:348] T 6b56adc64eba4898b80687a2679b70c8 P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } }
I20240502 01:53:35.732157   307 raft_consensus.cc:374] T 6b56adc64eba4898b80687a2679b70c8 P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:35.732240   307 raft_consensus.cc:729] T 6b56adc64eba4898b80687a2679b70c8 P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b191086bf5f2432b8d240510817ae5fe, State: Initialized, Role: FOLLOWER
I20240502 01:53:35.732465   307 consensus_queue.cc:260] T 6b56adc64eba4898b80687a2679b70c8 P b191086bf5f2432b8d240510817ae5fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } }
I20240502 01:53:35.733079   307 ts_tablet_manager.cc:1398] T 6b56adc64eba4898b80687a2679b70c8 P b191086bf5f2432b8d240510817ae5fe: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:53:35.744639   309 tablet_bootstrap.cc:492] T 6b56adc64eba4898b80687a2679b70c8 P 35f356f20e274db7bf020f78f1bc9ae7: No bootstrap required, opened a new log
I20240502 01:53:35.744850   309 ts_tablet_manager.cc:1367] T 6b56adc64eba4898b80687a2679b70c8 P 35f356f20e274db7bf020f78f1bc9ae7: Time spent bootstrapping tablet: real 0.033s	user 0.010s	sys 0.014s
I20240502 01:53:35.744951   306 tablet_bootstrap.cc:492] T 5575decad4be448e835db06b570c69a8 P 1864f9ea44c54742b090c594a5be3669: No bootstrap required, opened a new log
I20240502 01:53:35.745126   306 ts_tablet_manager.cc:1367] T 5575decad4be448e835db06b570c69a8 P 1864f9ea44c54742b090c594a5be3669: Time spent bootstrapping tablet: real 0.019s	user 0.005s	sys 0.010s
I20240502 01:53:35.745849   306 raft_consensus.cc:348] T 5575decad4be448e835db06b570c69a8 P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } }
I20240502 01:53:35.745801   309 raft_consensus.cc:348] T 6b56adc64eba4898b80687a2679b70c8 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } }
I20240502 01:53:35.746043   309 raft_consensus.cc:374] T 6b56adc64eba4898b80687a2679b70c8 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:35.746062   306 raft_consensus.cc:374] T 5575decad4be448e835db06b570c69a8 P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:35.746125   306 raft_consensus.cc:729] T 5575decad4be448e835db06b570c69a8 P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1864f9ea44c54742b090c594a5be3669, State: Initialized, Role: FOLLOWER
I20240502 01:53:35.746140   309 raft_consensus.cc:729] T 6b56adc64eba4898b80687a2679b70c8 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35f356f20e274db7bf020f78f1bc9ae7, State: Initialized, Role: FOLLOWER
I20240502 01:53:35.746423   306 consensus_queue.cc:260] T 5575decad4be448e835db06b570c69a8 P 1864f9ea44c54742b090c594a5be3669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } }
I20240502 01:53:35.746395   309 consensus_queue.cc:260] T 6b56adc64eba4898b80687a2679b70c8 P 35f356f20e274db7bf020f78f1bc9ae7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } }
I20240502 01:53:35.746994   306 ts_tablet_manager.cc:1398] T 5575decad4be448e835db06b570c69a8 P 1864f9ea44c54742b090c594a5be3669: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:53:35.748049   309 ts_tablet_manager.cc:1398] T 6b56adc64eba4898b80687a2679b70c8 P 35f356f20e274db7bf020f78f1bc9ae7: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 01:53:35.847992 32639 raft_consensus.cc:482] T 5575decad4be448e835db06b570c69a8 P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:35.848242 32639 raft_consensus.cc:504] T 5575decad4be448e835db06b570c69a8 P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } }
I20240502 01:53:35.848802 32639 leader_election.cc:288] T 5575decad4be448e835db06b570c69a8 P 1864f9ea44c54742b090c594a5be3669 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b191086bf5f2432b8d240510817ae5fe (127.25.109.129:42243)
I20240502 01:53:35.849452 31649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5575decad4be448e835db06b570c69a8" candidate_uuid: "1864f9ea44c54742b090c594a5be3669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b191086bf5f2432b8d240510817ae5fe" is_pre_election: true
I20240502 01:53:35.849717 31649 raft_consensus.cc:2454] T 5575decad4be448e835db06b570c69a8 P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1864f9ea44c54742b090c594a5be3669 in term 0.
I20240502 01:53:35.850231 31850 leader_election.cc:302] T 5575decad4be448e835db06b570c69a8 P 1864f9ea44c54742b090c594a5be3669 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1864f9ea44c54742b090c594a5be3669, b191086bf5f2432b8d240510817ae5fe; no voters: 
I20240502 01:53:35.850608 32639 raft_consensus.cc:2781] T 5575decad4be448e835db06b570c69a8 P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:35.850703 32639 raft_consensus.cc:482] T 5575decad4be448e835db06b570c69a8 P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:35.850814 32639 raft_consensus.cc:3037] T 5575decad4be448e835db06b570c69a8 P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:35.852787 32639 raft_consensus.cc:504] T 5575decad4be448e835db06b570c69a8 P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } }
I20240502 01:53:35.853298 32639 leader_election.cc:288] T 5575decad4be448e835db06b570c69a8 P 1864f9ea44c54742b090c594a5be3669 [CANDIDATE]: Term 1 election: Requested vote from peers b191086bf5f2432b8d240510817ae5fe (127.25.109.129:42243)
I20240502 01:53:35.853873 31649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5575decad4be448e835db06b570c69a8" candidate_uuid: "1864f9ea44c54742b090c594a5be3669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b191086bf5f2432b8d240510817ae5fe"
I20240502 01:53:35.854076 31649 raft_consensus.cc:3037] T 5575decad4be448e835db06b570c69a8 P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:35.855976 31649 raft_consensus.cc:2454] T 5575decad4be448e835db06b570c69a8 P b191086bf5f2432b8d240510817ae5fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1864f9ea44c54742b090c594a5be3669 in term 1.
I20240502 01:53:35.856413 31850 leader_election.cc:302] T 5575decad4be448e835db06b570c69a8 P 1864f9ea44c54742b090c594a5be3669 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1864f9ea44c54742b090c594a5be3669, b191086bf5f2432b8d240510817ae5fe; no voters: 
I20240502 01:53:35.856698 32639 raft_consensus.cc:2781] T 5575decad4be448e835db06b570c69a8 P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:35.857098 32639 raft_consensus.cc:686] T 5575decad4be448e835db06b570c69a8 P 1864f9ea44c54742b090c594a5be3669 [term 1 LEADER]: Becoming Leader. State: Replica: 1864f9ea44c54742b090c594a5be3669, State: Running, Role: LEADER
I20240502 01:53:35.857378 32639 consensus_queue.cc:237] T 5575decad4be448e835db06b570c69a8 P 1864f9ea44c54742b090c594a5be3669 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } }
I20240502 01:53:35.859972 31505 catalog_manager.cc:5484] T 5575decad4be448e835db06b570c69a8 P 1864f9ea44c54742b090c594a5be3669 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1864f9ea44c54742b090c594a5be3669 (127.25.109.131). New cstate: current_term: 1 leader_uuid: "1864f9ea44c54742b090c594a5be3669" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } }
I20240502 01:53:35.945787 32689 raft_consensus.cc:482] T 6b56adc64eba4898b80687a2679b70c8 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:35.945974 32689 raft_consensus.cc:504] T 6b56adc64eba4898b80687a2679b70c8 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } }
I20240502 01:53:35.946537 32689 leader_election.cc:288] T 6b56adc64eba4898b80687a2679b70c8 P 35f356f20e274db7bf020f78f1bc9ae7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b191086bf5f2432b8d240510817ae5fe (127.25.109.129:42243)
I20240502 01:53:35.947240 31649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b56adc64eba4898b80687a2679b70c8" candidate_uuid: "35f356f20e274db7bf020f78f1bc9ae7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b191086bf5f2432b8d240510817ae5fe" is_pre_election: true
I20240502 01:53:35.947471 31649 raft_consensus.cc:2454] T 6b56adc64eba4898b80687a2679b70c8 P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35f356f20e274db7bf020f78f1bc9ae7 in term 0.
I20240502 01:53:35.947993 31717 leader_election.cc:302] T 6b56adc64eba4898b80687a2679b70c8 P 35f356f20e274db7bf020f78f1bc9ae7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 35f356f20e274db7bf020f78f1bc9ae7, b191086bf5f2432b8d240510817ae5fe; no voters: 
I20240502 01:53:35.948360 32689 raft_consensus.cc:2781] T 6b56adc64eba4898b80687a2679b70c8 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:35.948493 32689 raft_consensus.cc:482] T 6b56adc64eba4898b80687a2679b70c8 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:35.948603 32689 raft_consensus.cc:3037] T 6b56adc64eba4898b80687a2679b70c8 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:35.950774 32689 raft_consensus.cc:504] T 6b56adc64eba4898b80687a2679b70c8 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } }
I20240502 01:53:35.951377 32689 leader_election.cc:288] T 6b56adc64eba4898b80687a2679b70c8 P 35f356f20e274db7bf020f78f1bc9ae7 [CANDIDATE]: Term 1 election: Requested vote from peers b191086bf5f2432b8d240510817ae5fe (127.25.109.129:42243)
I20240502 01:53:35.951967 31649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b56adc64eba4898b80687a2679b70c8" candidate_uuid: "35f356f20e274db7bf020f78f1bc9ae7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b191086bf5f2432b8d240510817ae5fe"
I20240502 01:53:35.952203 31649 raft_consensus.cc:3037] T 6b56adc64eba4898b80687a2679b70c8 P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:35.954134 31649 raft_consensus.cc:2454] T 6b56adc64eba4898b80687a2679b70c8 P b191086bf5f2432b8d240510817ae5fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35f356f20e274db7bf020f78f1bc9ae7 in term 1.
I20240502 01:53:35.954531 31717 leader_election.cc:302] T 6b56adc64eba4898b80687a2679b70c8 P 35f356f20e274db7bf020f78f1bc9ae7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 35f356f20e274db7bf020f78f1bc9ae7, b191086bf5f2432b8d240510817ae5fe; no voters: 
I20240502 01:53:35.954823 32689 raft_consensus.cc:2781] T 6b56adc64eba4898b80687a2679b70c8 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:35.954958 32689 raft_consensus.cc:686] T 6b56adc64eba4898b80687a2679b70c8 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 LEADER]: Becoming Leader. State: Replica: 35f356f20e274db7bf020f78f1bc9ae7, State: Running, Role: LEADER
I20240502 01:53:35.955214 32689 consensus_queue.cc:237] T 6b56adc64eba4898b80687a2679b70c8 P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } }
I20240502 01:53:35.957825 31505 catalog_manager.cc:5484] T 6b56adc64eba4898b80687a2679b70c8 P 35f356f20e274db7bf020f78f1bc9ae7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130). New cstate: current_term: 1 leader_uuid: "35f356f20e274db7bf020f78f1bc9ae7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } }
I20240502 01:53:36.058079 32639 raft_consensus.cc:482] T 1313391ae5b2440d93445deda2642a14 P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:36.058281 32639 raft_consensus.cc:504] T 1313391ae5b2440d93445deda2642a14 P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } }
I20240502 01:53:36.058830 32639 leader_election.cc:288] T 1313391ae5b2440d93445deda2642a14 P 1864f9ea44c54742b090c594a5be3669 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b191086bf5f2432b8d240510817ae5fe (127.25.109.129:42243)
I20240502 01:53:36.059495 31649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1313391ae5b2440d93445deda2642a14" candidate_uuid: "1864f9ea44c54742b090c594a5be3669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b191086bf5f2432b8d240510817ae5fe" is_pre_election: true
I20240502 01:53:36.059736 31649 raft_consensus.cc:2454] T 1313391ae5b2440d93445deda2642a14 P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1864f9ea44c54742b090c594a5be3669 in term 0.
I20240502 01:53:36.060209 31850 leader_election.cc:302] T 1313391ae5b2440d93445deda2642a14 P 1864f9ea44c54742b090c594a5be3669 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1864f9ea44c54742b090c594a5be3669, b191086bf5f2432b8d240510817ae5fe; no voters: 
I20240502 01:53:36.060494 32639 raft_consensus.cc:2781] T 1313391ae5b2440d93445deda2642a14 P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:36.060622 32639 raft_consensus.cc:482] T 1313391ae5b2440d93445deda2642a14 P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:36.060706 32639 raft_consensus.cc:3037] T 1313391ae5b2440d93445deda2642a14 P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:36.062739 32639 raft_consensus.cc:504] T 1313391ae5b2440d93445deda2642a14 P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } }
I20240502 01:53:36.063264 32639 leader_election.cc:288] T 1313391ae5b2440d93445deda2642a14 P 1864f9ea44c54742b090c594a5be3669 [CANDIDATE]: Term 1 election: Requested vote from peers b191086bf5f2432b8d240510817ae5fe (127.25.109.129:42243)
I20240502 01:53:36.063905 31649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1313391ae5b2440d93445deda2642a14" candidate_uuid: "1864f9ea44c54742b090c594a5be3669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b191086bf5f2432b8d240510817ae5fe"
I20240502 01:53:36.064143 31649 raft_consensus.cc:3037] T 1313391ae5b2440d93445deda2642a14 P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:36.066401 31649 raft_consensus.cc:2454] T 1313391ae5b2440d93445deda2642a14 P b191086bf5f2432b8d240510817ae5fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1864f9ea44c54742b090c594a5be3669 in term 1.
I20240502 01:53:36.066862 31850 leader_election.cc:302] T 1313391ae5b2440d93445deda2642a14 P 1864f9ea44c54742b090c594a5be3669 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1864f9ea44c54742b090c594a5be3669, b191086bf5f2432b8d240510817ae5fe; no voters: 
I20240502 01:53:36.067179 32639 raft_consensus.cc:2781] T 1313391ae5b2440d93445deda2642a14 P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:36.067333 32639 raft_consensus.cc:686] T 1313391ae5b2440d93445deda2642a14 P 1864f9ea44c54742b090c594a5be3669 [term 1 LEADER]: Becoming Leader. State: Replica: 1864f9ea44c54742b090c594a5be3669, State: Running, Role: LEADER
I20240502 01:53:36.067580 32639 consensus_queue.cc:237] T 1313391ae5b2440d93445deda2642a14 P 1864f9ea44c54742b090c594a5be3669 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } }
I20240502 01:53:36.070322 31505 catalog_manager.cc:5484] T 1313391ae5b2440d93445deda2642a14 P 1864f9ea44c54742b090c594a5be3669 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1864f9ea44c54742b090c594a5be3669 (127.25.109.131). New cstate: current_term: 1 leader_uuid: "1864f9ea44c54742b090c594a5be3669" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } }
I20240502 01:53:36.328256 32639 consensus_queue.cc:1035] T 5575decad4be448e835db06b570c69a8 P 1864f9ea44c54742b090c594a5be3669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:36.406396   320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:36.406747   320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:36.414753   320 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240502 01:53:36.501345 32689 consensus_queue.cc:1035] T 6b56adc64eba4898b80687a2679b70c8 P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:36.559368 31649 raft_consensus.cc:1261] T 1313391ae5b2440d93445deda2642a14 P b191086bf5f2432b8d240510817ae5fe [term 1 FOLLOWER]: Refusing update from remote peer 1864f9ea44c54742b090c594a5be3669: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:36.560216   315 consensus_queue.cc:1035] T 1313391ae5b2440d93445deda2642a14 P 1864f9ea44c54742b090c594a5be3669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 167.179 ms
  time per row: 0.0835893 ms
I20240502 01:53:36.723824 31505 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60436:
name: "rebalance_test_table_4"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:53:36.733731 31763 tablet_service.cc:1467] Processing CreateTablet for tablet 82e5853a509c43df917f257186844ddc (DEFAULT_TABLE table=rebalance_test_table_4 [id=9a09fee1a01c4c1c836c0dab06832248]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:36.734071 31896 tablet_service.cc:1467] Processing CreateTablet for tablet 82e5853a509c43df917f257186844ddc (DEFAULT_TABLE table=rebalance_test_table_4 [id=9a09fee1a01c4c1c836c0dab06832248]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:36.734230 31763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82e5853a509c43df917f257186844ddc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:36.734349 31628 tablet_service.cc:1467] Processing CreateTablet for tablet 4a00f35083c9404ea2ea877826f2fe77 (DEFAULT_TABLE table=rebalance_test_table_4 [id=9a09fee1a01c4c1c836c0dab06832248]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:36.734510 31896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82e5853a509c43df917f257186844ddc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:36.734767 31628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a00f35083c9404ea2ea877826f2fe77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:36.740592 31895 tablet_service.cc:1467] Processing CreateTablet for tablet e714cba54201433caa218073097c1e6a (DEFAULT_TABLE table=rebalance_test_table_4 [id=9a09fee1a01c4c1c836c0dab06832248]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:36.741045 31895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e714cba54201433caa218073097c1e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:36.743678 31762 tablet_service.cc:1467] Processing CreateTablet for tablet 4a00f35083c9404ea2ea877826f2fe77 (DEFAULT_TABLE table=rebalance_test_table_4 [id=9a09fee1a01c4c1c836c0dab06832248]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:36.744163 31762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a00f35083c9404ea2ea877826f2fe77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:36.745330 31764 tablet_service.cc:1467] Processing CreateTablet for tablet e714cba54201433caa218073097c1e6a (DEFAULT_TABLE table=rebalance_test_table_4 [id=9a09fee1a01c4c1c836c0dab06832248]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:53:36.745832 31764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e714cba54201433caa218073097c1e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:36.753278   373 tablet_bootstrap.cc:492] T 82e5853a509c43df917f257186844ddc P 35f356f20e274db7bf020f78f1bc9ae7: Bootstrap starting.
I20240502 01:53:36.755450   373 tablet_bootstrap.cc:654] T 82e5853a509c43df917f257186844ddc P 35f356f20e274db7bf020f78f1bc9ae7: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:36.757253   375 tablet_bootstrap.cc:492] T 82e5853a509c43df917f257186844ddc P 1864f9ea44c54742b090c594a5be3669: Bootstrap starting.
I20240502 01:53:36.758699   375 tablet_bootstrap.cc:654] T 82e5853a509c43df917f257186844ddc P 1864f9ea44c54742b090c594a5be3669: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:36.763381   374 tablet_bootstrap.cc:492] T 4a00f35083c9404ea2ea877826f2fe77 P b191086bf5f2432b8d240510817ae5fe: Bootstrap starting.
I20240502 01:53:36.767108   375 tablet_bootstrap.cc:492] T 82e5853a509c43df917f257186844ddc P 1864f9ea44c54742b090c594a5be3669: No bootstrap required, opened a new log
I20240502 01:53:36.767401   375 ts_tablet_manager.cc:1367] T 82e5853a509c43df917f257186844ddc P 1864f9ea44c54742b090c594a5be3669: Time spent bootstrapping tablet: real 0.010s	user 0.009s	sys 0.000s
I20240502 01:53:36.767923   374 tablet_bootstrap.cc:654] T 4a00f35083c9404ea2ea877826f2fe77 P b191086bf5f2432b8d240510817ae5fe: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:36.768435   375 raft_consensus.cc:348] T 82e5853a509c43df917f257186844ddc P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } }
I20240502 01:53:36.768733   375 raft_consensus.cc:374] T 82e5853a509c43df917f257186844ddc P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:36.768884   375 raft_consensus.cc:729] T 82e5853a509c43df917f257186844ddc P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1864f9ea44c54742b090c594a5be3669, State: Initialized, Role: FOLLOWER
I20240502 01:53:36.769229   375 consensus_queue.cc:260] T 82e5853a509c43df917f257186844ddc P 1864f9ea44c54742b090c594a5be3669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } }
I20240502 01:53:36.771211   375 ts_tablet_manager.cc:1398] T 82e5853a509c43df917f257186844ddc P 1864f9ea44c54742b090c594a5be3669: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 01:53:36.771577   375 tablet_bootstrap.cc:492] T e714cba54201433caa218073097c1e6a P 1864f9ea44c54742b090c594a5be3669: Bootstrap starting.
I20240502 01:53:36.773756   375 tablet_bootstrap.cc:654] T e714cba54201433caa218073097c1e6a P 1864f9ea44c54742b090c594a5be3669: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:36.776355   373 tablet_bootstrap.cc:492] T 82e5853a509c43df917f257186844ddc P 35f356f20e274db7bf020f78f1bc9ae7: No bootstrap required, opened a new log
I20240502 01:53:36.776531   373 ts_tablet_manager.cc:1367] T 82e5853a509c43df917f257186844ddc P 35f356f20e274db7bf020f78f1bc9ae7: Time spent bootstrapping tablet: real 0.023s	user 0.005s	sys 0.010s
I20240502 01:53:36.777067   375 tablet_bootstrap.cc:492] T e714cba54201433caa218073097c1e6a P 1864f9ea44c54742b090c594a5be3669: No bootstrap required, opened a new log
I20240502 01:53:36.777289   375 ts_tablet_manager.cc:1367] T e714cba54201433caa218073097c1e6a P 1864f9ea44c54742b090c594a5be3669: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.000s
I20240502 01:53:36.778190   375 raft_consensus.cc:348] T e714cba54201433caa218073097c1e6a P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } }
I20240502 01:53:36.778498   375 raft_consensus.cc:374] T e714cba54201433caa218073097c1e6a P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:36.778663   375 raft_consensus.cc:729] T e714cba54201433caa218073097c1e6a P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1864f9ea44c54742b090c594a5be3669, State: Initialized, Role: FOLLOWER
I20240502 01:53:36.778985   375 consensus_queue.cc:260] T e714cba54201433caa218073097c1e6a P 1864f9ea44c54742b090c594a5be3669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } }
I20240502 01:53:36.779192   374 tablet_bootstrap.cc:492] T 4a00f35083c9404ea2ea877826f2fe77 P b191086bf5f2432b8d240510817ae5fe: No bootstrap required, opened a new log
I20240502 01:53:36.779368   374 ts_tablet_manager.cc:1367] T 4a00f35083c9404ea2ea877826f2fe77 P b191086bf5f2432b8d240510817ae5fe: Time spent bootstrapping tablet: real 0.016s	user 0.007s	sys 0.000s
I20240502 01:53:36.779448   373 raft_consensus.cc:348] T 82e5853a509c43df917f257186844ddc P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } }
I20240502 01:53:36.779690   373 raft_consensus.cc:374] T 82e5853a509c43df917f257186844ddc P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:36.779847   373 raft_consensus.cc:729] T 82e5853a509c43df917f257186844ddc P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35f356f20e274db7bf020f78f1bc9ae7, State: Initialized, Role: FOLLOWER
I20240502 01:53:36.780118   373 consensus_queue.cc:260] T 82e5853a509c43df917f257186844ddc P 35f356f20e274db7bf020f78f1bc9ae7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } }
I20240502 01:53:36.780230   374 raft_consensus.cc:348] T 4a00f35083c9404ea2ea877826f2fe77 P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } }
I20240502 01:53:36.780440   374 raft_consensus.cc:374] T 4a00f35083c9404ea2ea877826f2fe77 P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:36.780534   374 raft_consensus.cc:729] T 4a00f35083c9404ea2ea877826f2fe77 P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b191086bf5f2432b8d240510817ae5fe, State: Initialized, Role: FOLLOWER
I20240502 01:53:36.780539   375 ts_tablet_manager.cc:1398] T e714cba54201433caa218073097c1e6a P 1864f9ea44c54742b090c594a5be3669: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:53:36.780782   374 consensus_queue.cc:260] T 4a00f35083c9404ea2ea877826f2fe77 P b191086bf5f2432b8d240510817ae5fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } }
I20240502 01:53:36.781459   374 ts_tablet_manager.cc:1398] T 4a00f35083c9404ea2ea877826f2fe77 P b191086bf5f2432b8d240510817ae5fe: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:53:36.783399   373 ts_tablet_manager.cc:1398] T 82e5853a509c43df917f257186844ddc P 35f356f20e274db7bf020f78f1bc9ae7: Time spent starting tablet: real 0.007s	user 0.000s	sys 0.003s
I20240502 01:53:36.783761   373 tablet_bootstrap.cc:492] T 4a00f35083c9404ea2ea877826f2fe77 P 35f356f20e274db7bf020f78f1bc9ae7: Bootstrap starting.
I20240502 01:53:36.785425   373 tablet_bootstrap.cc:654] T 4a00f35083c9404ea2ea877826f2fe77 P 35f356f20e274db7bf020f78f1bc9ae7: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:36.787285   373 tablet_bootstrap.cc:492] T 4a00f35083c9404ea2ea877826f2fe77 P 35f356f20e274db7bf020f78f1bc9ae7: No bootstrap required, opened a new log
I20240502 01:53:36.787427   373 ts_tablet_manager.cc:1367] T 4a00f35083c9404ea2ea877826f2fe77 P 35f356f20e274db7bf020f78f1bc9ae7: Time spent bootstrapping tablet: real 0.004s	user 0.002s	sys 0.001s
I20240502 01:53:36.788070   373 raft_consensus.cc:348] T 4a00f35083c9404ea2ea877826f2fe77 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } }
I20240502 01:53:36.788244   373 raft_consensus.cc:374] T 4a00f35083c9404ea2ea877826f2fe77 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:36.788316   373 raft_consensus.cc:729] T 4a00f35083c9404ea2ea877826f2fe77 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35f356f20e274db7bf020f78f1bc9ae7, State: Initialized, Role: FOLLOWER
I20240502 01:53:36.788528   373 consensus_queue.cc:260] T 4a00f35083c9404ea2ea877826f2fe77 P 35f356f20e274db7bf020f78f1bc9ae7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } }
I20240502 01:53:36.789005   373 ts_tablet_manager.cc:1398] T 4a00f35083c9404ea2ea877826f2fe77 P 35f356f20e274db7bf020f78f1bc9ae7: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 01:53:36.789326   373 tablet_bootstrap.cc:492] T e714cba54201433caa218073097c1e6a P 35f356f20e274db7bf020f78f1bc9ae7: Bootstrap starting.
I20240502 01:53:36.790896   373 tablet_bootstrap.cc:654] T e714cba54201433caa218073097c1e6a P 35f356f20e274db7bf020f78f1bc9ae7: Neither blocks nor log segments found. Creating new log.
I20240502 01:53:36.792831   373 tablet_bootstrap.cc:492] T e714cba54201433caa218073097c1e6a P 35f356f20e274db7bf020f78f1bc9ae7: No bootstrap required, opened a new log
I20240502 01:53:36.792989   373 ts_tablet_manager.cc:1367] T e714cba54201433caa218073097c1e6a P 35f356f20e274db7bf020f78f1bc9ae7: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.002s
I20240502 01:53:36.793728   373 raft_consensus.cc:348] T e714cba54201433caa218073097c1e6a P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } }
I20240502 01:53:36.793920   373 raft_consensus.cc:374] T e714cba54201433caa218073097c1e6a P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:53:36.794011   373 raft_consensus.cc:729] T e714cba54201433caa218073097c1e6a P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35f356f20e274db7bf020f78f1bc9ae7, State: Initialized, Role: FOLLOWER
I20240502 01:53:36.794216   373 consensus_queue.cc:260] T e714cba54201433caa218073097c1e6a P 35f356f20e274db7bf020f78f1bc9ae7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } }
I20240502 01:53:36.794652   373 ts_tablet_manager.cc:1398] T e714cba54201433caa218073097c1e6a P 35f356f20e274db7bf020f78f1bc9ae7: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:53:37.035976 32689 raft_consensus.cc:482] T e714cba54201433caa218073097c1e6a P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:37.036164 32689 raft_consensus.cc:504] T e714cba54201433caa218073097c1e6a P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } }
I20240502 01:53:37.036680 32689 leader_election.cc:288] T e714cba54201433caa218073097c1e6a P 35f356f20e274db7bf020f78f1bc9ae7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1864f9ea44c54742b090c594a5be3669 (127.25.109.131:42119)
I20240502 01:53:37.037256 31916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e714cba54201433caa218073097c1e6a" candidate_uuid: "35f356f20e274db7bf020f78f1bc9ae7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1864f9ea44c54742b090c594a5be3669" is_pre_election: true
I20240502 01:53:37.037499 31916 raft_consensus.cc:2454] T e714cba54201433caa218073097c1e6a P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35f356f20e274db7bf020f78f1bc9ae7 in term 0.
I20240502 01:53:37.037940 31716 leader_election.cc:302] T e714cba54201433caa218073097c1e6a P 35f356f20e274db7bf020f78f1bc9ae7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1864f9ea44c54742b090c594a5be3669, 35f356f20e274db7bf020f78f1bc9ae7; no voters: 
I20240502 01:53:37.038216 32689 raft_consensus.cc:2781] T e714cba54201433caa218073097c1e6a P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:37.038323 32689 raft_consensus.cc:482] T e714cba54201433caa218073097c1e6a P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:37.038403 32689 raft_consensus.cc:3037] T e714cba54201433caa218073097c1e6a P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:37.040299 32689 raft_consensus.cc:504] T e714cba54201433caa218073097c1e6a P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } }
I20240502 01:53:37.040742 32689 leader_election.cc:288] T e714cba54201433caa218073097c1e6a P 35f356f20e274db7bf020f78f1bc9ae7 [CANDIDATE]: Term 1 election: Requested vote from peers 1864f9ea44c54742b090c594a5be3669 (127.25.109.131:42119)
I20240502 01:53:37.041242 31916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e714cba54201433caa218073097c1e6a" candidate_uuid: "35f356f20e274db7bf020f78f1bc9ae7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1864f9ea44c54742b090c594a5be3669"
I20240502 01:53:37.041437 31916 raft_consensus.cc:3037] T e714cba54201433caa218073097c1e6a P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:37.043344 31916 raft_consensus.cc:2454] T e714cba54201433caa218073097c1e6a P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35f356f20e274db7bf020f78f1bc9ae7 in term 1.
I20240502 01:53:37.043751 31716 leader_election.cc:302] T e714cba54201433caa218073097c1e6a P 35f356f20e274db7bf020f78f1bc9ae7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1864f9ea44c54742b090c594a5be3669, 35f356f20e274db7bf020f78f1bc9ae7; no voters: 
I20240502 01:53:37.044054 32689 raft_consensus.cc:2781] T e714cba54201433caa218073097c1e6a P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:37.044191 32689 raft_consensus.cc:686] T e714cba54201433caa218073097c1e6a P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 LEADER]: Becoming Leader. State: Replica: 35f356f20e274db7bf020f78f1bc9ae7, State: Running, Role: LEADER
I20240502 01:53:37.044498 32689 consensus_queue.cc:237] T e714cba54201433caa218073097c1e6a P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } }
I20240502 01:53:37.047171 31497 catalog_manager.cc:5484] T e714cba54201433caa218073097c1e6a P 35f356f20e274db7bf020f78f1bc9ae7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130). New cstate: current_term: 1 leader_uuid: "35f356f20e274db7bf020f78f1bc9ae7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } }
I20240502 01:53:37.055370 32689 raft_consensus.cc:482] T 4a00f35083c9404ea2ea877826f2fe77 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:37.055595 32689 raft_consensus.cc:504] T 4a00f35083c9404ea2ea877826f2fe77 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } }
I20240502 01:53:37.056171 32689 leader_election.cc:288] T 4a00f35083c9404ea2ea877826f2fe77 P 35f356f20e274db7bf020f78f1bc9ae7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b191086bf5f2432b8d240510817ae5fe (127.25.109.129:42243)
I20240502 01:53:37.056831 31648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a00f35083c9404ea2ea877826f2fe77" candidate_uuid: "35f356f20e274db7bf020f78f1bc9ae7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b191086bf5f2432b8d240510817ae5fe" is_pre_election: true
I20240502 01:53:37.057045 31648 raft_consensus.cc:2454] T 4a00f35083c9404ea2ea877826f2fe77 P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35f356f20e274db7bf020f78f1bc9ae7 in term 0.
I20240502 01:53:37.057446 31717 leader_election.cc:302] T 4a00f35083c9404ea2ea877826f2fe77 P 35f356f20e274db7bf020f78f1bc9ae7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 35f356f20e274db7bf020f78f1bc9ae7, b191086bf5f2432b8d240510817ae5fe; no voters: 
I20240502 01:53:37.057716 32689 raft_consensus.cc:2781] T 4a00f35083c9404ea2ea877826f2fe77 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:37.057823 32689 raft_consensus.cc:482] T 4a00f35083c9404ea2ea877826f2fe77 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:37.057929 32689 raft_consensus.cc:3037] T 4a00f35083c9404ea2ea877826f2fe77 P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:37.059846 32689 raft_consensus.cc:504] T 4a00f35083c9404ea2ea877826f2fe77 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } }
I20240502 01:53:37.060447 32689 leader_election.cc:288] T 4a00f35083c9404ea2ea877826f2fe77 P 35f356f20e274db7bf020f78f1bc9ae7 [CANDIDATE]: Term 1 election: Requested vote from peers b191086bf5f2432b8d240510817ae5fe (127.25.109.129:42243)
I20240502 01:53:37.060930 31648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a00f35083c9404ea2ea877826f2fe77" candidate_uuid: "35f356f20e274db7bf020f78f1bc9ae7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b191086bf5f2432b8d240510817ae5fe"
I20240502 01:53:37.061110 31648 raft_consensus.cc:3037] T 4a00f35083c9404ea2ea877826f2fe77 P b191086bf5f2432b8d240510817ae5fe [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:37.063079 31648 raft_consensus.cc:2454] T 4a00f35083c9404ea2ea877826f2fe77 P b191086bf5f2432b8d240510817ae5fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35f356f20e274db7bf020f78f1bc9ae7 in term 1.
I20240502 01:53:37.063491 31717 leader_election.cc:302] T 4a00f35083c9404ea2ea877826f2fe77 P 35f356f20e274db7bf020f78f1bc9ae7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 35f356f20e274db7bf020f78f1bc9ae7, b191086bf5f2432b8d240510817ae5fe; no voters: 
I20240502 01:53:37.063763 32689 raft_consensus.cc:2781] T 4a00f35083c9404ea2ea877826f2fe77 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:37.063964 32689 raft_consensus.cc:686] T 4a00f35083c9404ea2ea877826f2fe77 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 LEADER]: Becoming Leader. State: Replica: 35f356f20e274db7bf020f78f1bc9ae7, State: Running, Role: LEADER
I20240502 01:53:37.064214 32689 consensus_queue.cc:237] T 4a00f35083c9404ea2ea877826f2fe77 P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } }
I20240502 01:53:37.066779 31497 catalog_manager.cc:5484] T 4a00f35083c9404ea2ea877826f2fe77 P 35f356f20e274db7bf020f78f1bc9ae7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130). New cstate: current_term: 1 leader_uuid: "35f356f20e274db7bf020f78f1bc9ae7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } }
I20240502 01:53:37.145963 32689 raft_consensus.cc:482] T 82e5853a509c43df917f257186844ddc P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:37.146157 32689 raft_consensus.cc:504] T 82e5853a509c43df917f257186844ddc P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } }
I20240502 01:53:37.146739 32689 leader_election.cc:288] T 82e5853a509c43df917f257186844ddc P 35f356f20e274db7bf020f78f1bc9ae7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1864f9ea44c54742b090c594a5be3669 (127.25.109.131:42119)
I20240502 01:53:37.147308 31916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82e5853a509c43df917f257186844ddc" candidate_uuid: "35f356f20e274db7bf020f78f1bc9ae7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1864f9ea44c54742b090c594a5be3669" is_pre_election: true
I20240502 01:53:37.147526 31916 raft_consensus.cc:2454] T 82e5853a509c43df917f257186844ddc P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35f356f20e274db7bf020f78f1bc9ae7 in term 0.
I20240502 01:53:37.147969 31716 leader_election.cc:302] T 82e5853a509c43df917f257186844ddc P 35f356f20e274db7bf020f78f1bc9ae7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1864f9ea44c54742b090c594a5be3669, 35f356f20e274db7bf020f78f1bc9ae7; no voters: 
I20240502 01:53:37.148252 32689 raft_consensus.cc:2781] T 82e5853a509c43df917f257186844ddc P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:53:37.148362 32689 raft_consensus.cc:482] T 82e5853a509c43df917f257186844ddc P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:37.148445 32689 raft_consensus.cc:3037] T 82e5853a509c43df917f257186844ddc P 35f356f20e274db7bf020f78f1bc9ae7 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:37.150362 32689 raft_consensus.cc:504] T 82e5853a509c43df917f257186844ddc P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } }
I20240502 01:53:37.150827 32689 leader_election.cc:288] T 82e5853a509c43df917f257186844ddc P 35f356f20e274db7bf020f78f1bc9ae7 [CANDIDATE]: Term 1 election: Requested vote from peers 1864f9ea44c54742b090c594a5be3669 (127.25.109.131:42119)
I20240502 01:53:37.151342 31916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82e5853a509c43df917f257186844ddc" candidate_uuid: "35f356f20e274db7bf020f78f1bc9ae7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1864f9ea44c54742b090c594a5be3669"
I20240502 01:53:37.151525 31916 raft_consensus.cc:3037] T 82e5853a509c43df917f257186844ddc P 1864f9ea44c54742b090c594a5be3669 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:53:37.153350 31916 raft_consensus.cc:2454] T 82e5853a509c43df917f257186844ddc P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35f356f20e274db7bf020f78f1bc9ae7 in term 1.
I20240502 01:53:37.153743 31716 leader_election.cc:302] T 82e5853a509c43df917f257186844ddc P 35f356f20e274db7bf020f78f1bc9ae7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1864f9ea44c54742b090c594a5be3669, 35f356f20e274db7bf020f78f1bc9ae7; no voters: 
I20240502 01:53:37.154023 32689 raft_consensus.cc:2781] T 82e5853a509c43df917f257186844ddc P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:53:37.154158 32689 raft_consensus.cc:686] T 82e5853a509c43df917f257186844ddc P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 LEADER]: Becoming Leader. State: Replica: 35f356f20e274db7bf020f78f1bc9ae7, State: Running, Role: LEADER
I20240502 01:53:37.154455 32689 consensus_queue.cc:237] T 82e5853a509c43df917f257186844ddc P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } }
I20240502 01:53:37.157083 31497 catalog_manager.cc:5484] T 82e5853a509c43df917f257186844ddc P 35f356f20e274db7bf020f78f1bc9ae7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130). New cstate: current_term: 1 leader_uuid: "35f356f20e274db7bf020f78f1bc9ae7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } }
I20240502 01:53:37.475147 32689 consensus_queue.cc:1035] T 4a00f35083c9404ea2ea877826f2fe77 P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:53:37.542353   385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:37.542686   385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:37.550522   385 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:53:37.568434   348 consensus_queue.cc:1035] T 82e5853a509c43df917f257186844ddc P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_4'
I20240502 01:53:37.595157   348 consensus_queue.cc:1035] T e714cba54201433caa218073097c1e6a P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 151.039 ms
  time per row: 0.0755196 ms
I20240502 01:53:37.856854 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.132:34649
--local_ip_for_outbound_sockets=127.25.109.132
--tserver_master_addrs=127.25.109.190:41417
--webserver_port=38497
--webserver_interface=127.25.109.132
--builtin_ntp_servers=127.25.109.148:40813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 01:53:38.192384   433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:38.192698   433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:38.192838   433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:38.201229   433 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:53:38.201393   433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:38.201553   433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.132
I20240502 01:53:38.212054   433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:40813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.132:34649
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.25.109.132
--webserver_port=38497
--tserver_master_addrs=127.25.109.190:41417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.132
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:38.213441   433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:38.215286   433 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:38.229072   438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:38.229597   439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:38.230091   441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:38.230700   433 server_base.cc:1034] running on GCE node
I20240502 01:53:38.231338   433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:38.232429   433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:38.233693   433 hybrid_clock.cc:648] HybridClock initialized: now 1714614818233622 us; error 50 us; skew 500 ppm
I20240502 01:53:38.234068   433 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:38.242945   433 webserver.cc:460] Webserver started at http://127.25.109.132:38497/ using document root <none> and password file <none>
I20240502 01:53:38.243827   433 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:38.244006   433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:38.249716   433 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.004s	sys 0.000s
I20240502 01:53:38.252902   447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:38.254434   433 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20240502 01:53:38.254590   433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "0aea20ce2a1144519ed144383c016d78"
format_stamp: "Formatted at 2024-05-02 01:53:24 on dist-test-slave-kj4b"
I20240502 01:53:38.255496   433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:38.298691   433 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:53:38.299741   433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:38.300119   433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:38.301317   433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:38.303251   433 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:38.303344   433 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:38.303474   433 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:38.303529   433 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:38.342483   433 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.132:34649
I20240502 01:53:38.342540   561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.132:34649 every 8 connection(s)
I20240502 01:53:38.344080   433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 01:53:38.346154 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 433
I20240502 01:53:38.346832 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.133:46379
--local_ip_for_outbound_sockets=127.25.109.133
--tserver_master_addrs=127.25.109.190:41417
--webserver_port=45597
--webserver_interface=127.25.109.133
--builtin_ntp_servers=127.25.109.148:40813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:53:38.364647   562 heartbeater.cc:347] Connected to a master server at 127.25.109.190:41417
I20240502 01:53:38.365069   562 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:38.366180   562 heartbeater.cc:511] Master 127.25.109.190:41417 requested a full tablet report, sending...
I20240502 01:53:38.368147 31501 ts_manager.cc:194] Re-registered known tserver with Master: 0aea20ce2a1144519ed144383c016d78 (127.25.109.132:34649)
I20240502 01:53:38.369163 31501 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.132:39121
W20240502 01:53:38.681512   566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:38.681833   566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:38.681962   566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:38.689823   566 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:53:38.689985   566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:38.690140   566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.133
I20240502 01:53:38.699633   566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:40813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.133:46379
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.25.109.133
--webserver_port=45597
--tserver_master_addrs=127.25.109.190:41417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.133
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:38.701122   566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:38.702925   566 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:38.716234   572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:38.716706   574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:38.717156   571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:38.717784   566 server_base.cc:1034] running on GCE node
I20240502 01:53:38.718410   566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:38.719369   566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:38.720558   566 hybrid_clock.cc:648] HybridClock initialized: now 1714614818720500 us; error 45 us; skew 500 ppm
I20240502 01:53:38.720877   566 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:38.729400   566 webserver.cc:460] Webserver started at http://127.25.109.133:45597/ using document root <none> and password file <none>
I20240502 01:53:38.730391   566 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:38.730549   566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:38.735976   566 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.002s	sys 0.001s
I20240502 01:53:38.738874   580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:38.740236   566 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.002s
I20240502 01:53:38.740386   566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "a506969c1a5346c09c19a01f6cbc1a8b"
format_stamp: "Formatted at 2024-05-02 01:53:25 on dist-test-slave-kj4b"
I20240502 01:53:38.741231   566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:38.770351   566 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:53:38.771497   566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:38.771904   566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:38.773139   566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:38.776449   587 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240502 01:53:38.782191   566 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240502 01:53:38.782325   566 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.006s	user 0.000s	sys 0.000s
I20240502 01:53:38.782460   566 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240502 01:53:38.785470   566 ts_tablet_manager.cc:590] Registered 1 tablets
I20240502 01:53:38.785593   566 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:53:38.785934   587 tablet_bootstrap.cc:492] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b: Bootstrap starting.
I20240502 01:53:38.804821   587 log.cc:824] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:38.808722   587 tablet_bootstrap.cc:492] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 01:53:38.809705   587 tablet_bootstrap.cc:492] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b: Bootstrap complete.
I20240502 01:53:38.811172   587 ts_tablet_manager.cc:1367] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b: Time spent bootstrapping tablet: real 0.025s	user 0.013s	sys 0.009s
I20240502 01:53:38.821362   587 raft_consensus.cc:348] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } }
I20240502 01:53:38.822887   587 raft_consensus.cc:729] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a506969c1a5346c09c19a01f6cbc1a8b, State: Initialized, Role: FOLLOWER
I20240502 01:53:38.823715   587 consensus_queue.cc:260] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } }
I20240502 01:53:38.825608   587 ts_tablet_manager.cc:1398] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.007s
I20240502 01:53:38.833428   566 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.133:46379
I20240502 01:53:38.833498   698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.133:46379 every 8 connection(s)
I20240502 01:53:38.834766   566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data/info.pb
W20240502 01:53:38.835944   700 tablet.cc:2369] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:53:38.837199 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 566
I20240502 01:53:38.837929 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.134:38633
--local_ip_for_outbound_sockets=127.25.109.134
--tserver_master_addrs=127.25.109.190:41417
--webserver_port=39535
--webserver_interface=127.25.109.134
--builtin_ntp_servers=127.25.109.148:40813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:53:38.846765   699 heartbeater.cc:347] Connected to a master server at 127.25.109.190:41417
I20240502 01:53:38.847067   699 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:38.847792   699 heartbeater.cc:511] Master 127.25.109.190:41417 requested a full tablet report, sending...
I20240502 01:53:38.850085 31501 ts_manager.cc:194] Re-registered known tserver with Master: a506969c1a5346c09c19a01f6cbc1a8b (127.25.109.133:46379)
I20240502 01:53:38.851238 31501 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.133:40955
W20240502 01:53:39.174124   702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:39.174443   702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:39.174571   702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:39.182783   702 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:53:39.182960   702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:39.183173   702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.134
I20240502 01:53:39.193076   702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:40813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.134:38633
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.25.109.134
--webserver_port=39535
--tserver_master_addrs=127.25.109.190:41417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.134
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:39.194398   702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:39.196230   702 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:39.208730   708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:39.209259   711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:39.209720   709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:39.210299   702 server_base.cc:1034] running on GCE node
I20240502 01:53:39.210937   702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:39.211938   702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:39.213156   702 hybrid_clock.cc:648] HybridClock initialized: now 1714614819213086 us; error 53 us; skew 500 ppm
I20240502 01:53:39.213503   702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:39.215766   702 webserver.cc:460] Webserver started at http://127.25.109.134:39535/ using document root <none> and password file <none>
I20240502 01:53:39.216564   702 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:39.216697   702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:39.222307   702 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.002s	sys 0.003s
I20240502 01:53:39.225286   717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:39.226681   702 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 01:53:39.226861   702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "676092c80dd74040bb49a06db1ac209c"
format_stamp: "Formatted at 2024-05-02 01:53:25 on dist-test-slave-kj4b"
I20240502 01:53:39.227690   702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:39.249073   702 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:53:39.249961   702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:39.250259   702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:39.251251   702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:39.252915   702 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:39.252996   702 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:39.253123   702 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:39.253176   702 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:39.284591   702 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.134:38633
I20240502 01:53:39.284607   832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.134:38633 every 8 connection(s)
I20240502 01:53:39.286178   702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 01:53:39.287777 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 702
I20240502 01:53:39.288545 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.135:37253
--local_ip_for_outbound_sockets=127.25.109.135
--tserver_master_addrs=127.25.109.190:41417
--webserver_port=39311
--webserver_interface=127.25.109.135
--builtin_ntp_servers=127.25.109.148:40813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:53:39.301874   833 heartbeater.cc:347] Connected to a master server at 127.25.109.190:41417
I20240502 01:53:39.302204   833 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:39.303004   833 heartbeater.cc:511] Master 127.25.109.190:41417 requested a full tablet report, sending...
I20240502 01:53:39.304500 31501 ts_manager.cc:194] Re-registered known tserver with Master: 676092c80dd74040bb49a06db1ac209c (127.25.109.134:38633)
I20240502 01:53:39.305320 31501 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.134:41075
I20240502 01:53:39.372522   562 heartbeater.cc:503] Master 127.25.109.190:41417 was elected leader, sending a full tablet report...
W20240502 01:53:39.634670   836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:39.634989   836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:39.635097   836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:39.642724   836 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:53:39.642851   836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:39.642972   836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.135
I20240502 01:53:39.652379   836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:40813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.135:37253
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.25.109.135
--webserver_port=39311
--tserver_master_addrs=127.25.109.190:41417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.135
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:39.653682   836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:39.655457   836 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:39.667898   841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:39.668442   842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:39.668825   844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:39.669447   836 server_base.cc:1034] running on GCE node
I20240502 01:53:39.670091   836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:39.671056   836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:39.672240   836 hybrid_clock.cc:648] HybridClock initialized: now 1714614819672181 us; error 55 us; skew 500 ppm
I20240502 01:53:39.672547   836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:39.675071   836 webserver.cc:460] Webserver started at http://127.25.109.135:39311/ using document root <none> and password file <none>
I20240502 01:53:39.676079   836 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:39.676210   836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:39.681828   836 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240502 01:53:39.684687   850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:39.686043   836 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 01:53:39.686198   836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "facf1046c8054d2d841cf875c63e29e2"
format_stamp: "Formatted at 2024-05-02 01:53:25 on dist-test-slave-kj4b"
I20240502 01:53:39.687027   836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:39.721518   836 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:53:39.722395   836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:39.722689   836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:39.723690   836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:39.725590   836 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:39.725674   836 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:39.725803   836 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:39.725857   836 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:53:39.758013   836 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.135:37253
I20240502 01:53:39.758085   964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.135:37253 every 8 connection(s)
I20240502 01:53:39.759539   836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 01:53:39.761317 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 836
I20240502 01:53:39.761613 26038 test_util.cc:268] Using random seed: -549311065
I20240502 01:53:39.774482   965 heartbeater.cc:347] Connected to a master server at 127.25.109.190:41417
I20240502 01:53:39.774865   965 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:39.775699   965 heartbeater.cc:511] Master 127.25.109.190:41417 requested a full tablet report, sending...
I20240502 01:53:39.777338 31501 ts_manager.cc:194] Re-registered known tserver with Master: facf1046c8054d2d841cf875c63e29e2 (127.25.109.135:37253)
I20240502 01:53:39.778303 31501 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.135:36039
I20240502 01:53:39.784497 26038 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240502 01:53:39.784970 26038 test_util.cc:268] Using random seed: -549287705
I20240502 01:53:39.802003 26038 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240502 01:53:39.802429 26038 test_util.cc:268] Using random seed: -549270264
I20240502 01:53:39.827535 26038 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240502 01:53:39.832250 26038 test_util.cc:268] Using random seed: -549240469
I20240502 01:53:39.854225   699 heartbeater.cc:503] Master 127.25.109.190:41417 was elected leader, sending a full tablet report...
I20240502 01:53:39.916201 26038 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240502 01:53:39.939052 26038 test_util.cc:268] Using random seed: -549133638
I20240502 01:53:39.987232 26038 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
I20240502 01:53:40.100180  1063 raft_consensus.cc:482] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:53:40.100616  1063 raft_consensus.cc:504] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } }
I20240502 01:53:40.114127  1063 leader_election.cc:288] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1864f9ea44c54742b090c594a5be3669 (127.25.109.131:42119)
I20240502 01:53:40.143307 31916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a865d3891d4e4430a0a0aec06a074cea" candidate_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1864f9ea44c54742b090c594a5be3669" is_pre_election: true
I20240502 01:53:40.143599 31916 raft_consensus.cc:2454] T a865d3891d4e4430a0a0aec06a074cea P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a506969c1a5346c09c19a01f6cbc1a8b in term 1.
I20240502 01:53:40.150067   581 leader_election.cc:302] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1864f9ea44c54742b090c594a5be3669, a506969c1a5346c09c19a01f6cbc1a8b; no voters: 
I20240502 01:53:40.150800  1063 raft_consensus.cc:2781] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:53:40.150976  1063 raft_consensus.cc:482] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:53:40.151073  1063 raft_consensus.cc:3037] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:40.157159  1063 raft_consensus.cc:504] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } }
I20240502 01:53:40.158502  1063 leader_election.cc:288] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b [CANDIDATE]: Term 2 election: Requested vote from peers 1864f9ea44c54742b090c594a5be3669 (127.25.109.131:42119)
I20240502 01:53:40.177063 31914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a865d3891d4e4430a0a0aec06a074cea" candidate_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1864f9ea44c54742b090c594a5be3669"
I20240502 01:53:40.177382 31914 raft_consensus.cc:3037] T a865d3891d4e4430a0a0aec06a074cea P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:40.193527 31914 raft_consensus.cc:2454] T a865d3891d4e4430a0a0aec06a074cea P 1864f9ea44c54742b090c594a5be3669 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a506969c1a5346c09c19a01f6cbc1a8b in term 2.
I20240502 01:53:40.194039   581 leader_election.cc:302] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1864f9ea44c54742b090c594a5be3669, a506969c1a5346c09c19a01f6cbc1a8b; no voters: 
I20240502 01:53:40.194500  1063 raft_consensus.cc:2781] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:53:40.206988  1063 raft_consensus.cc:686] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b [term 2 LEADER]: Becoming Leader. State: Replica: a506969c1a5346c09c19a01f6cbc1a8b, State: Running, Role: LEADER
I20240502 01:53:40.207559  1063 consensus_queue.cc:237] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } }
I20240502 01:53:40.244598 31501 catalog_manager.cc:5484] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } }
I20240502 01:53:40.307804   833 heartbeater.cc:503] Master 127.25.109.190:41417 was elected leader, sending a full tablet report...
I20240502 01:53:40.781142   965 heartbeater.cc:503] Master 127.25.109.190:41417 was elected leader, sending a full tablet report...
I20240502 01:53:40.790153 31915 raft_consensus.cc:1261] T a865d3891d4e4430a0a0aec06a074cea P 1864f9ea44c54742b090c594a5be3669 [term 2 FOLLOWER]: Refusing update from remote peer a506969c1a5346c09c19a01f6cbc1a8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 01:53:40.791069  1093 consensus_queue.cc:1035] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:53:41.478219   497 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:41.486239   630 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:41.513499 31896 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:53:41.542829   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:41.551538 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:41.585180   766 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:41.609030 31764 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:53:42.470490   497 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:42.704175 31764 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:53:42.729841 31896 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:53:42.765980   630 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:42.786442   766 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:42.806141 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:42.885267   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:43.202121 31913 consensus_queue.cc:237] T 1313391ae5b2440d93445deda2642a14 P 1864f9ea44c54742b090c594a5be3669 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 1.49, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } }
I20240502 01:53:43.206681 31649 raft_consensus.cc:1261] T 1313391ae5b2440d93445deda2642a14 P b191086bf5f2432b8d240510817ae5fe [term 1 FOLLOWER]: Refusing update from remote peer 1864f9ea44c54742b090c594a5be3669: Log matching property violated. Preceding OpId in replica: term: 1 index: 49. Preceding OpId from leader: term: 1 index: 50. (index mismatch)
I20240502 01:53:43.207350  1151 consensus_queue.cc:1035] T 1313391ae5b2440d93445deda2642a14 P 1864f9ea44c54742b090c594a5be3669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
W20240502 01:53:43.228334 31852 consensus_peers.cc:485] T 1313391ae5b2440d93445deda2642a14 P 1864f9ea44c54742b090c594a5be3669 -> Peer facf1046c8054d2d841cf875c63e29e2 (127.25.109.135:37253): Couldn't send request to peer facf1046c8054d2d841cf875c63e29e2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1313391ae5b2440d93445deda2642a14. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:43.240250  1151 raft_consensus.cc:2932] T 1313391ae5b2440d93445deda2642a14 P 1864f9ea44c54742b090c594a5be3669 [term 1 LEADER]: Committing config change with OpId 1.50: config changed from index -1 to 50, VOTER facf1046c8054d2d841cf875c63e29e2 (127.25.109.135) added. New config: { opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } } }
I20240502 01:53:43.257375 31499 catalog_manager.cc:5484] T 1313391ae5b2440d93445deda2642a14 P 1864f9ea44c54742b090c594a5be3669 reported cstate change: config changed from index -1 to 50, VOTER facf1046c8054d2d841cf875c63e29e2 (127.25.109.135) added. New cstate: current_term: 1 leader_uuid: "1864f9ea44c54742b090c594a5be3669" committed_config { opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } } }
I20240502 01:53:43.269960 31645 raft_consensus.cc:2932] T 1313391ae5b2440d93445deda2642a14 P b191086bf5f2432b8d240510817ae5fe [term 1 FOLLOWER]: Committing config change with OpId 1.50: config changed from index -1 to 50, VOTER facf1046c8054d2d841cf875c63e29e2 (127.25.109.135) added. New config: { opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } } }
I20240502 01:53:43.593097 31784 consensus_queue.cc:237] T 4a00f35083c9404ea2ea877826f2fe77 P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 66, Committed index: 66, Last appended: 1.67, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } }
I20240502 01:53:43.597175 31647 raft_consensus.cc:1261] T 4a00f35083c9404ea2ea877826f2fe77 P b191086bf5f2432b8d240510817ae5fe [term 1 FOLLOWER]: Refusing update from remote peer 35f356f20e274db7bf020f78f1bc9ae7: Log matching property violated. Preceding OpId in replica: term: 1 index: 67. Preceding OpId from leader: term: 1 index: 68. (index mismatch)
I20240502 01:53:43.598340  1045 consensus_queue.cc:1035] T 4a00f35083c9404ea2ea877826f2fe77 P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 68, Last known committed idx: 66, Time since last communication: 0.004s
I20240502 01:53:43.605567  1045 raft_consensus.cc:2932] T 4a00f35083c9404ea2ea877826f2fe77 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 LEADER]: Committing config change with OpId 1.68: config changed from index -1 to 68, VOTER 676092c80dd74040bb49a06db1ac209c (127.25.109.134) added. New config: { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } } }
I20240502 01:53:43.613301 31645 raft_consensus.cc:2932] T 4a00f35083c9404ea2ea877826f2fe77 P b191086bf5f2432b8d240510817ae5fe [term 1 FOLLOWER]: Committing config change with OpId 1.68: config changed from index -1 to 68, VOTER 676092c80dd74040bb49a06db1ac209c (127.25.109.134) added. New config: { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } } }
I20240502 01:53:43.613853 31500 catalog_manager.cc:5484] T 4a00f35083c9404ea2ea877826f2fe77 P 35f356f20e274db7bf020f78f1bc9ae7 reported cstate change: config changed from index -1 to 68, VOTER 676092c80dd74040bb49a06db1ac209c (127.25.109.134) added. New cstate: current_term: 1 leader_uuid: "35f356f20e274db7bf020f78f1bc9ae7" committed_config { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } } }
W20240502 01:53:43.642462 31719 consensus_peers.cc:485] T 4a00f35083c9404ea2ea877826f2fe77 P 35f356f20e274db7bf020f78f1bc9ae7 -> Peer 676092c80dd74040bb49a06db1ac209c (127.25.109.134:38633): Couldn't send request to peer 676092c80dd74040bb49a06db1ac209c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4a00f35083c9404ea2ea877826f2fe77. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:43.855399  1257 ts_tablet_manager.cc:902] T 1313391ae5b2440d93445deda2642a14 P facf1046c8054d2d841cf875c63e29e2: Initiating tablet copy from peer 1864f9ea44c54742b090c594a5be3669 (127.25.109.131:42119)
I20240502 01:53:43.915405 31784 consensus_queue.cc:237] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 63, Committed index: 63, Last appended: 1.63, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } }
I20240502 01:53:43.918429 31913 raft_consensus.cc:1261] T 238e56f8d83748e8b2cf1100e92db4c6 P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Refusing update from remote peer 35f356f20e274db7bf020f78f1bc9ae7: Log matching property violated. Preceding OpId in replica: term: 1 index: 63. Preceding OpId from leader: term: 1 index: 64. (index mismatch)
I20240502 01:53:43.919039   995 consensus_queue.cc:1035] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 64, Last known committed idx: 63, Time since last communication: 0.000s
I20240502 01:53:43.925197  1046 raft_consensus.cc:2932] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 LEADER]: Committing config change with OpId 1.64: config changed from index -1 to 64, VOTER 0aea20ce2a1144519ed144383c016d78 (127.25.109.132) added. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } }
I20240502 01:53:43.932250 31500 catalog_manager.cc:5484] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7 reported cstate change: config changed from index -1 to 64, VOTER 0aea20ce2a1144519ed144383c016d78 (127.25.109.132) added. New cstate: current_term: 1 leader_uuid: "35f356f20e274db7bf020f78f1bc9ae7" committed_config { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } }
I20240502 01:53:43.941148 31915 raft_consensus.cc:2932] T 238e56f8d83748e8b2cf1100e92db4c6 P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Committing config change with OpId 1.64: config changed from index -1 to 64, VOTER 0aea20ce2a1144519ed144383c016d78 (127.25.109.132) added. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } }
W20240502 01:53:43.950513 31716 consensus_peers.cc:485] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7 -> Peer 0aea20ce2a1144519ed144383c016d78 (127.25.109.132:34649): Couldn't send request to peer 0aea20ce2a1144519ed144383c016d78. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 238e56f8d83748e8b2cf1100e92db4c6. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:43.995229  1257 tablet_copy_client.cc:308] T 1313391ae5b2440d93445deda2642a14 P facf1046c8054d2d841cf875c63e29e2: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.131:42119
I20240502 01:53:44.006234  1264 ts_tablet_manager.cc:902] T 4a00f35083c9404ea2ea877826f2fe77 P 676092c80dd74040bb49a06db1ac209c: Initiating tablet copy from peer 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130:41639)
I20240502 01:53:44.109665  1264 tablet_copy_client.cc:308] T 4a00f35083c9404ea2ea877826f2fe77 P 676092c80dd74040bb49a06db1ac209c: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.130:41639
I20240502 01:53:44.137374 31936 tablet_copy_service.cc:140] P 1864f9ea44c54742b090c594a5be3669: Received BeginTabletCopySession request for tablet 1313391ae5b2440d93445deda2642a14 from peer facf1046c8054d2d841cf875c63e29e2 ({username='slave'} at 127.25.109.135:36921)
I20240502 01:53:44.137713 31936 tablet_copy_service.cc:161] P 1864f9ea44c54742b090c594a5be3669: Beginning new tablet copy session on tablet 1313391ae5b2440d93445deda2642a14 from peer facf1046c8054d2d841cf875c63e29e2 at {username='slave'} at 127.25.109.135:36921: session id = facf1046c8054d2d841cf875c63e29e2-1313391ae5b2440d93445deda2642a14
I20240502 01:53:44.137639 31805 tablet_copy_service.cc:140] P 35f356f20e274db7bf020f78f1bc9ae7: Received BeginTabletCopySession request for tablet 4a00f35083c9404ea2ea877826f2fe77 from peer 676092c80dd74040bb49a06db1ac209c ({username='slave'} at 127.25.109.134:45991)
I20240502 01:53:44.137948 31805 tablet_copy_service.cc:161] P 35f356f20e274db7bf020f78f1bc9ae7: Beginning new tablet copy session on tablet 4a00f35083c9404ea2ea877826f2fe77 from peer 676092c80dd74040bb49a06db1ac209c at {username='slave'} at 127.25.109.134:45991: session id = 676092c80dd74040bb49a06db1ac209c-4a00f35083c9404ea2ea877826f2fe77
I20240502 01:53:44.146224 31936 tablet_copy_source_session.cc:205] T 1313391ae5b2440d93445deda2642a14 P 1864f9ea44c54742b090c594a5be3669: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:44.153102  1257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1313391ae5b2440d93445deda2642a14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:44.158445 31805 tablet_copy_source_session.cc:205] T 4a00f35083c9404ea2ea877826f2fe77 P 35f356f20e274db7bf020f78f1bc9ae7: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:44.172536  1264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a00f35083c9404ea2ea877826f2fe77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:44.187600  1257 tablet_copy_client.cc:783] T 1313391ae5b2440d93445deda2642a14 P facf1046c8054d2d841cf875c63e29e2: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:44.188071  1257 tablet_copy_client.cc:647] T 1313391ae5b2440d93445deda2642a14 P facf1046c8054d2d841cf875c63e29e2: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:44.199069  1257 tablet_copy_client.cc:523] T 1313391ae5b2440d93445deda2642a14 P facf1046c8054d2d841cf875c63e29e2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:44.204802  1257 tablet_bootstrap.cc:492] T 1313391ae5b2440d93445deda2642a14 P facf1046c8054d2d841cf875c63e29e2: Bootstrap starting.
I20240502 01:53:44.209031  1264 tablet_copy_client.cc:783] T 4a00f35083c9404ea2ea877826f2fe77 P 676092c80dd74040bb49a06db1ac209c: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:44.216544  1264 tablet_copy_client.cc:647] T 4a00f35083c9404ea2ea877826f2fe77 P 676092c80dd74040bb49a06db1ac209c: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:44.233925  1257 log.cc:824] T 1313391ae5b2440d93445deda2642a14 P facf1046c8054d2d841cf875c63e29e2: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:44.259846  1264 tablet_copy_client.cc:523] T 4a00f35083c9404ea2ea877826f2fe77 P 676092c80dd74040bb49a06db1ac209c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:44.265038  1264 tablet_bootstrap.cc:492] T 4a00f35083c9404ea2ea877826f2fe77 P 676092c80dd74040bb49a06db1ac209c: Bootstrap starting.
I20240502 01:53:44.321375  1264 log.cc:824] T 4a00f35083c9404ea2ea877826f2fe77 P 676092c80dd74040bb49a06db1ac209c: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:44.424667  1290 ts_tablet_manager.cc:902] T 238e56f8d83748e8b2cf1100e92db4c6 P 0aea20ce2a1144519ed144383c016d78: Initiating tablet copy from peer 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130:41639)
I20240502 01:53:44.550319  1264 tablet_bootstrap.cc:492] T 4a00f35083c9404ea2ea877826f2fe77 P 676092c80dd74040bb49a06db1ac209c: Bootstrap replayed 1/1 log segments. Stats: ops{read=76 overwritten=0 applied=76 ignored=0} inserts{seen=748 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:44.551157  1264 tablet_bootstrap.cc:492] T 4a00f35083c9404ea2ea877826f2fe77 P 676092c80dd74040bb49a06db1ac209c: Bootstrap complete.
I20240502 01:53:44.551939  1264 ts_tablet_manager.cc:1367] T 4a00f35083c9404ea2ea877826f2fe77 P 676092c80dd74040bb49a06db1ac209c: Time spent bootstrapping tablet: real 0.287s	user 0.159s	sys 0.017s
I20240502 01:53:44.559438  1264 raft_consensus.cc:348] T 4a00f35083c9404ea2ea877826f2fe77 P 676092c80dd74040bb49a06db1ac209c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } }
I20240502 01:53:44.560173  1264 raft_consensus.cc:729] T 4a00f35083c9404ea2ea877826f2fe77 P 676092c80dd74040bb49a06db1ac209c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 676092c80dd74040bb49a06db1ac209c, State: Initialized, Role: FOLLOWER
I20240502 01:53:44.561196  1264 consensus_queue.cc:260] T 4a00f35083c9404ea2ea877826f2fe77 P 676092c80dd74040bb49a06db1ac209c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 76, Last appended: 1.76, Last appended by leader: 76, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } }
I20240502 01:53:44.591264  1264 ts_tablet_manager.cc:1398] T 4a00f35083c9404ea2ea877826f2fe77 P 676092c80dd74040bb49a06db1ac209c: Time spent starting tablet: real 0.039s	user 0.017s	sys 0.019s
I20240502 01:53:44.594918 31805 tablet_copy_service.cc:342] P 35f356f20e274db7bf020f78f1bc9ae7: Request end of tablet copy session 676092c80dd74040bb49a06db1ac209c-4a00f35083c9404ea2ea877826f2fe77 received from {username='slave'} at 127.25.109.134:45991
I20240502 01:53:44.595145 31805 tablet_copy_service.cc:433] P 35f356f20e274db7bf020f78f1bc9ae7: ending tablet copy session 676092c80dd74040bb49a06db1ac209c-4a00f35083c9404ea2ea877826f2fe77 on tablet 4a00f35083c9404ea2ea877826f2fe77 with peer 676092c80dd74040bb49a06db1ac209c
I20240502 01:53:44.603471   786 raft_consensus.cc:1203] T 4a00f35083c9404ea2ea877826f2fe77 P 676092c80dd74040bb49a06db1ac209c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.67->[1.68-1.80]   Dedup: 1.76->[1.77-1.80]
I20240502 01:53:44.612828  1257 tablet_bootstrap.cc:492] T 1313391ae5b2440d93445deda2642a14 P facf1046c8054d2d841cf875c63e29e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=66 overwritten=0 applied=66 ignored=0} inserts{seen=751 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:44.614167  1257 tablet_bootstrap.cc:492] T 1313391ae5b2440d93445deda2642a14 P facf1046c8054d2d841cf875c63e29e2: Bootstrap complete.
I20240502 01:53:44.614774  1257 ts_tablet_manager.cc:1367] T 1313391ae5b2440d93445deda2642a14 P facf1046c8054d2d841cf875c63e29e2: Time spent bootstrapping tablet: real 0.410s	user 0.148s	sys 0.011s
I20240502 01:53:44.635756 31915 consensus_queue.cc:237] T 5575decad4be448e835db06b570c69a8 P 1864f9ea44c54742b090c594a5be3669 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 99, Committed index: 99, Last appended: 1.99, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } }
I20240502 01:53:44.640532  1257 raft_consensus.cc:348] T 1313391ae5b2440d93445deda2642a14 P facf1046c8054d2d841cf875c63e29e2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } }
I20240502 01:53:44.642644  1257 raft_consensus.cc:729] T 1313391ae5b2440d93445deda2642a14 P facf1046c8054d2d841cf875c63e29e2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: facf1046c8054d2d841cf875c63e29e2, State: Initialized, Role: FOLLOWER
I20240502 01:53:44.644268  1257 consensus_queue.cc:260] T 1313391ae5b2440d93445deda2642a14 P facf1046c8054d2d841cf875c63e29e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 66, Last appended: 1.66, Last appended by leader: 66, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } }
I20240502 01:53:44.670020 31646 raft_consensus.cc:1261] T 5575decad4be448e835db06b570c69a8 P b191086bf5f2432b8d240510817ae5fe [term 1 FOLLOWER]: Refusing update from remote peer 1864f9ea44c54742b090c594a5be3669: Log matching property violated. Preceding OpId in replica: term: 1 index: 99. Preceding OpId from leader: term: 1 index: 100. (index mismatch)
I20240502 01:53:44.670868  1215 consensus_queue.cc:1035] T 5575decad4be448e835db06b570c69a8 P 1864f9ea44c54742b090c594a5be3669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 100, Last known committed idx: 99, Time since last communication: 0.000s
I20240502 01:53:44.674722  1215 raft_consensus.cc:2932] T 5575decad4be448e835db06b570c69a8 P 1864f9ea44c54742b090c594a5be3669 [term 1 LEADER]: Committing config change with OpId 1.100: config changed from index -1 to 100, VOTER a506969c1a5346c09c19a01f6cbc1a8b (127.25.109.133) added. New config: { opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } }
I20240502 01:53:44.677987 31648 raft_consensus.cc:2932] T 5575decad4be448e835db06b570c69a8 P b191086bf5f2432b8d240510817ae5fe [term 1 FOLLOWER]: Committing config change with OpId 1.100: config changed from index -1 to 100, VOTER a506969c1a5346c09c19a01f6cbc1a8b (127.25.109.133) added. New config: { opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } }
I20240502 01:53:44.684804  1257 ts_tablet_manager.cc:1398] T 1313391ae5b2440d93445deda2642a14 P facf1046c8054d2d841cf875c63e29e2: Time spent starting tablet: real 0.068s	user 0.012s	sys 0.010s
I20240502 01:53:44.687206 31500 catalog_manager.cc:5484] T 5575decad4be448e835db06b570c69a8 P b191086bf5f2432b8d240510817ae5fe reported cstate change: config changed from index -1 to 100, VOTER a506969c1a5346c09c19a01f6cbc1a8b (127.25.109.133) added. New cstate: current_term: 1 leader_uuid: "1864f9ea44c54742b090c594a5be3669" committed_config { opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } }
I20240502 01:53:44.713708 31936 tablet_copy_service.cc:342] P 1864f9ea44c54742b090c594a5be3669: Request end of tablet copy session facf1046c8054d2d841cf875c63e29e2-1313391ae5b2440d93445deda2642a14 received from {username='slave'} at 127.25.109.135:36921
I20240502 01:53:44.713973 31936 tablet_copy_service.cc:433] P 1864f9ea44c54742b090c594a5be3669: ending tablet copy session facf1046c8054d2d841cf875c63e29e2-1313391ae5b2440d93445deda2642a14 on tablet 1313391ae5b2440d93445deda2642a14 with peer facf1046c8054d2d841cf875c63e29e2
I20240502 01:53:44.766043   920 raft_consensus.cc:1203] T 1313391ae5b2440d93445deda2642a14 P facf1046c8054d2d841cf875c63e29e2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.49->[1.50-1.77]   Dedup: 1.66->[1.67-1.77]
I20240502 01:53:44.796794  1290 tablet_copy_client.cc:308] T 238e56f8d83748e8b2cf1100e92db4c6 P 0aea20ce2a1144519ed144383c016d78: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.130:41639
W20240502 01:53:44.840018 31852 consensus_peers.cc:485] T 5575decad4be448e835db06b570c69a8 P 1864f9ea44c54742b090c594a5be3669 -> Peer a506969c1a5346c09c19a01f6cbc1a8b (127.25.109.133:46379): Couldn't send request to peer a506969c1a5346c09c19a01f6cbc1a8b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5575decad4be448e835db06b570c69a8. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:44.880553 31805 tablet_copy_service.cc:140] P 35f356f20e274db7bf020f78f1bc9ae7: Received BeginTabletCopySession request for tablet 238e56f8d83748e8b2cf1100e92db4c6 from peer 0aea20ce2a1144519ed144383c016d78 ({username='slave'} at 127.25.109.132:56925)
I20240502 01:53:44.880760 31805 tablet_copy_service.cc:161] P 35f356f20e274db7bf020f78f1bc9ae7: Beginning new tablet copy session on tablet 238e56f8d83748e8b2cf1100e92db4c6 from peer 0aea20ce2a1144519ed144383c016d78 at {username='slave'} at 127.25.109.132:56925: session id = 0aea20ce2a1144519ed144383c016d78-238e56f8d83748e8b2cf1100e92db4c6
I20240502 01:53:44.883881 31805 tablet_copy_source_session.cc:205] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:44.896678  1290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 238e56f8d83748e8b2cf1100e92db4c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:44.953318  1290 tablet_copy_client.cc:783] T 238e56f8d83748e8b2cf1100e92db4c6 P 0aea20ce2a1144519ed144383c016d78: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:44.955528  1290 tablet_copy_client.cc:647] T 238e56f8d83748e8b2cf1100e92db4c6 P 0aea20ce2a1144519ed144383c016d78: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:44.964088  1290 tablet_copy_client.cc:523] T 238e56f8d83748e8b2cf1100e92db4c6 P 0aea20ce2a1144519ed144383c016d78: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:44.983230  1290 tablet_bootstrap.cc:492] T 238e56f8d83748e8b2cf1100e92db4c6 P 0aea20ce2a1144519ed144383c016d78: Bootstrap starting.
I20240502 01:53:45.049949  1290 log.cc:824] T 238e56f8d83748e8b2cf1100e92db4c6 P 0aea20ce2a1144519ed144383c016d78: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:45.115880  1330 ts_tablet_manager.cc:902] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b: Initiating tablet copy from peer 1864f9ea44c54742b090c594a5be3669 (127.25.109.131:42119)
I20240502 01:53:45.175737  1330 tablet_copy_client.cc:308] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.131:42119
I20240502 01:53:45.180063 31936 tablet_copy_service.cc:140] P 1864f9ea44c54742b090c594a5be3669: Received BeginTabletCopySession request for tablet 5575decad4be448e835db06b570c69a8 from peer a506969c1a5346c09c19a01f6cbc1a8b ({username='slave'} at 127.25.109.133:44189)
I20240502 01:53:45.180336 31936 tablet_copy_service.cc:161] P 1864f9ea44c54742b090c594a5be3669: Beginning new tablet copy session on tablet 5575decad4be448e835db06b570c69a8 from peer a506969c1a5346c09c19a01f6cbc1a8b at {username='slave'} at 127.25.109.133:44189: session id = a506969c1a5346c09c19a01f6cbc1a8b-5575decad4be448e835db06b570c69a8
I20240502 01:53:45.183357 31936 tablet_copy_source_session.cc:205] T 5575decad4be448e835db06b570c69a8 P 1864f9ea44c54742b090c594a5be3669: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:45.231546  1330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5575decad4be448e835db06b570c69a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:45.234306 31914 consensus_queue.cc:237] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 1.70, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } }
W20240502 01:53:45.241501 31852 consensus_peers.cc:485] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669 -> Peer facf1046c8054d2d841cf875c63e29e2 (127.25.109.135:37253): Couldn't send request to peer facf1046c8054d2d841cf875c63e29e2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 559610821917470fbcf5641d36acf409. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:45.242697 31647 raft_consensus.cc:1261] T 559610821917470fbcf5641d36acf409 P b191086bf5f2432b8d240510817ae5fe [term 1 FOLLOWER]: Refusing update from remote peer 1864f9ea44c54742b090c594a5be3669: Log matching property violated. Preceding OpId in replica: term: 1 index: 70. Preceding OpId from leader: term: 1 index: 71. (index mismatch)
I20240502 01:53:45.244513  1229 consensus_queue.cc:1035] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 71, Last known committed idx: 69, Time since last communication: 0.000s
I20240502 01:53:45.255071  1290 tablet_bootstrap.cc:492] T 238e56f8d83748e8b2cf1100e92db4c6 P 0aea20ce2a1144519ed144383c016d78: Bootstrap replayed 1/1 log segments. Stats: ops{read=77 overwritten=0 applied=77 ignored=0} inserts{seen=708 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:45.257375  1290 tablet_bootstrap.cc:492] T 238e56f8d83748e8b2cf1100e92db4c6 P 0aea20ce2a1144519ed144383c016d78: Bootstrap complete.
I20240502 01:53:45.258045  1290 ts_tablet_manager.cc:1367] T 238e56f8d83748e8b2cf1100e92db4c6 P 0aea20ce2a1144519ed144383c016d78: Time spent bootstrapping tablet: real 0.275s	user 0.156s	sys 0.005s
I20240502 01:53:45.265280  1290 raft_consensus.cc:348] T 238e56f8d83748e8b2cf1100e92db4c6 P 0aea20ce2a1144519ed144383c016d78 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } }
I20240502 01:53:45.265969  1290 raft_consensus.cc:729] T 238e56f8d83748e8b2cf1100e92db4c6 P 0aea20ce2a1144519ed144383c016d78 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aea20ce2a1144519ed144383c016d78, State: Initialized, Role: FOLLOWER
I20240502 01:53:45.266976  1290 consensus_queue.cc:260] T 238e56f8d83748e8b2cf1100e92db4c6 P 0aea20ce2a1144519ed144383c016d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 77, Last appended: 1.77, Last appended by leader: 77, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } }
I20240502 01:53:45.276348  1290 ts_tablet_manager.cc:1398] T 238e56f8d83748e8b2cf1100e92db4c6 P 0aea20ce2a1144519ed144383c016d78: Time spent starting tablet: real 0.018s	user 0.016s	sys 0.000s
I20240502 01:53:45.277889  1215 raft_consensus.cc:2932] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669 [term 1 LEADER]: Committing config change with OpId 1.71: config changed from index -1 to 71, VOTER facf1046c8054d2d841cf875c63e29e2 (127.25.109.135) added. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } } }
I20240502 01:53:45.284071 31805 tablet_copy_service.cc:342] P 35f356f20e274db7bf020f78f1bc9ae7: Request end of tablet copy session 0aea20ce2a1144519ed144383c016d78-238e56f8d83748e8b2cf1100e92db4c6 received from {username='slave'} at 127.25.109.132:56925
I20240502 01:53:45.284222 31805 tablet_copy_service.cc:433] P 35f356f20e274db7bf020f78f1bc9ae7: ending tablet copy session 0aea20ce2a1144519ed144383c016d78-238e56f8d83748e8b2cf1100e92db4c6 on tablet 238e56f8d83748e8b2cf1100e92db4c6 with peer 0aea20ce2a1144519ed144383c016d78
I20240502 01:53:45.288702 31500 catalog_manager.cc:5484] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669 reported cstate change: config changed from index -1 to 71, VOTER facf1046c8054d2d841cf875c63e29e2 (127.25.109.135) added. New cstate: current_term: 1 leader_uuid: "1864f9ea44c54742b090c594a5be3669" committed_config { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } } }
I20240502 01:53:45.293318 31644 raft_consensus.cc:2932] T 559610821917470fbcf5641d36acf409 P b191086bf5f2432b8d240510817ae5fe [term 1 FOLLOWER]: Committing config change with OpId 1.71: config changed from index -1 to 71, VOTER facf1046c8054d2d841cf875c63e29e2 (127.25.109.135) added. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } } }
I20240502 01:53:45.298940  1330 tablet_copy_client.cc:783] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:45.299453  1330 tablet_copy_client.cc:647] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:45.305457  1330 tablet_copy_client.cc:523] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:45.326923  1330 tablet_bootstrap.cc:492] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b: Bootstrap starting.
I20240502 01:53:45.458081   517 raft_consensus.cc:1203] T 238e56f8d83748e8b2cf1100e92db4c6 P 0aea20ce2a1144519ed144383c016d78 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.63->[1.64-1.84]   Dedup: 1.77->[1.78-1.84]
I20240502 01:53:45.679796  1330 tablet_bootstrap.cc:492] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b: Bootstrap replayed 1/1 log segments. Stats: ops{read=111 overwritten=0 applied=110 ignored=0} inserts{seen=741 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 01:53:45.680527  1330 tablet_bootstrap.cc:492] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b: Bootstrap complete.
I20240502 01:53:45.680902  1330 ts_tablet_manager.cc:1367] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b: Time spent bootstrapping tablet: real 0.354s	user 0.174s	sys 0.011s
I20240502 01:53:45.682150  1330 raft_consensus.cc:348] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } }
I20240502 01:53:45.704942  1330 raft_consensus.cc:729] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a506969c1a5346c09c19a01f6cbc1a8b, State: Initialized, Role: FOLLOWER
I20240502 01:53:45.705420  1330 consensus_queue.cc:260] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 110, Last appended: 1.111, Last appended by leader: 111, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } }
I20240502 01:53:45.709100  1330 ts_tablet_manager.cc:1398] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b: Time spent starting tablet: real 0.028s	user 0.017s	sys 0.007s
I20240502 01:53:45.713570 31936 tablet_copy_service.cc:342] P 1864f9ea44c54742b090c594a5be3669: Request end of tablet copy session a506969c1a5346c09c19a01f6cbc1a8b-5575decad4be448e835db06b570c69a8 received from {username='slave'} at 127.25.109.133:44189
I20240502 01:53:45.713727 31936 tablet_copy_service.cc:433] P 1864f9ea44c54742b090c594a5be3669: ending tablet copy session a506969c1a5346c09c19a01f6cbc1a8b-5575decad4be448e835db06b570c69a8 on tablet 5575decad4be448e835db06b570c69a8 with peer a506969c1a5346c09c19a01f6cbc1a8b
I20240502 01:53:45.855955  1371 ts_tablet_manager.cc:902] T 559610821917470fbcf5641d36acf409 P facf1046c8054d2d841cf875c63e29e2: Initiating tablet copy from peer 1864f9ea44c54742b090c594a5be3669 (127.25.109.131:42119)
I20240502 01:53:45.889279  1371 tablet_copy_client.cc:308] T 559610821917470fbcf5641d36acf409 P facf1046c8054d2d841cf875c63e29e2: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.131:42119
I20240502 01:53:45.890697 31936 tablet_copy_service.cc:140] P 1864f9ea44c54742b090c594a5be3669: Received BeginTabletCopySession request for tablet 559610821917470fbcf5641d36acf409 from peer facf1046c8054d2d841cf875c63e29e2 ({username='slave'} at 127.25.109.135:36921)
I20240502 01:53:45.890925 31936 tablet_copy_service.cc:161] P 1864f9ea44c54742b090c594a5be3669: Beginning new tablet copy session on tablet 559610821917470fbcf5641d36acf409 from peer facf1046c8054d2d841cf875c63e29e2 at {username='slave'} at 127.25.109.135:36921: session id = facf1046c8054d2d841cf875c63e29e2-559610821917470fbcf5641d36acf409
I20240502 01:53:45.893795 31936 tablet_copy_source_session.cc:205] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:45.895351  1371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 559610821917470fbcf5641d36acf409. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:45.901794 31644 consensus_queue.cc:237] T 86f2e43af7fb4e02bb853f0748d62ddb P b191086bf5f2432b8d240510817ae5fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 1.112, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } }
I20240502 01:53:45.915823  1371 tablet_copy_client.cc:783] T 559610821917470fbcf5641d36acf409 P facf1046c8054d2d841cf875c63e29e2: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:45.916260  1371 tablet_copy_client.cc:647] T 559610821917470fbcf5641d36acf409 P facf1046c8054d2d841cf875c63e29e2: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:45.919186  1371 tablet_copy_client.cc:523] T 559610821917470fbcf5641d36acf409 P facf1046c8054d2d841cf875c63e29e2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:45.922477 31784 raft_consensus.cc:1261] T 86f2e43af7fb4e02bb853f0748d62ddb P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 FOLLOWER]: Refusing update from remote peer b191086bf5f2432b8d240510817ae5fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 112. Preceding OpId from leader: term: 1 index: 113. (index mismatch)
I20240502 01:53:45.923169  1206 consensus_queue.cc:1035] T 86f2e43af7fb4e02bb853f0748d62ddb P b191086bf5f2432b8d240510817ae5fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 113, Last known committed idx: 112, Time since last communication: 0.000s
I20240502 01:53:45.925108  1371 tablet_bootstrap.cc:492] T 559610821917470fbcf5641d36acf409 P facf1046c8054d2d841cf875c63e29e2: Bootstrap starting.
I20240502 01:53:45.926442  1206 raft_consensus.cc:2932] T 86f2e43af7fb4e02bb853f0748d62ddb P b191086bf5f2432b8d240510817ae5fe [term 1 LEADER]: Committing config change with OpId 1.113: config changed from index -1 to 113, VOTER 0aea20ce2a1144519ed144383c016d78 (127.25.109.132) added. New config: { opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } }
I20240502 01:53:45.933192 31784 raft_consensus.cc:2932] T 86f2e43af7fb4e02bb853f0748d62ddb P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 FOLLOWER]: Committing config change with OpId 1.113: config changed from index -1 to 113, VOTER 0aea20ce2a1144519ed144383c016d78 (127.25.109.132) added. New config: { opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } }
I20240502 01:53:45.949467 31500 catalog_manager.cc:5484] T 86f2e43af7fb4e02bb853f0748d62ddb P b191086bf5f2432b8d240510817ae5fe reported cstate change: config changed from index -1 to 113, VOTER 0aea20ce2a1144519ed144383c016d78 (127.25.109.132) added. New cstate: current_term: 1 leader_uuid: "b191086bf5f2432b8d240510817ae5fe" committed_config { opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } }
I20240502 01:53:45.956040   652 raft_consensus.cc:1203] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.99->[1.100-1.124]   Dedup: 1.111->[1.112-1.124]
W20240502 01:53:45.974562 31581 consensus_peers.cc:485] T 86f2e43af7fb4e02bb853f0748d62ddb P b191086bf5f2432b8d240510817ae5fe -> Peer 0aea20ce2a1144519ed144383c016d78 (127.25.109.132:34649): Couldn't send request to peer 0aea20ce2a1144519ed144383c016d78. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 86f2e43af7fb4e02bb853f0748d62ddb. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:46.213059  1371 tablet_bootstrap.cc:492] T 559610821917470fbcf5641d36acf409 P facf1046c8054d2d841cf875c63e29e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=74 overwritten=0 applied=74 ignored=0} inserts{seen=746 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:46.214111  1371 tablet_bootstrap.cc:492] T 559610821917470fbcf5641d36acf409 P facf1046c8054d2d841cf875c63e29e2: Bootstrap complete.
I20240502 01:53:46.214509  1371 ts_tablet_manager.cc:1367] T 559610821917470fbcf5641d36acf409 P facf1046c8054d2d841cf875c63e29e2: Time spent bootstrapping tablet: real 0.289s	user 0.142s	sys 0.011s
I20240502 01:53:46.215607  1371 raft_consensus.cc:348] T 559610821917470fbcf5641d36acf409 P facf1046c8054d2d841cf875c63e29e2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } }
I20240502 01:53:46.215967  1371 raft_consensus.cc:729] T 559610821917470fbcf5641d36acf409 P facf1046c8054d2d841cf875c63e29e2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: facf1046c8054d2d841cf875c63e29e2, State: Initialized, Role: FOLLOWER
I20240502 01:53:46.216370  1371 consensus_queue.cc:260] T 559610821917470fbcf5641d36acf409 P facf1046c8054d2d841cf875c63e29e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 74, Last appended: 1.74, Last appended by leader: 74, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } }
I20240502 01:53:46.226090  1371 ts_tablet_manager.cc:1398] T 559610821917470fbcf5641d36acf409 P facf1046c8054d2d841cf875c63e29e2: Time spent starting tablet: real 0.011s	user 0.005s	sys 0.003s
I20240502 01:53:46.228245 31936 tablet_copy_service.cc:342] P 1864f9ea44c54742b090c594a5be3669: Request end of tablet copy session facf1046c8054d2d841cf875c63e29e2-559610821917470fbcf5641d36acf409 received from {username='slave'} at 127.25.109.135:36921
I20240502 01:53:46.228435 31936 tablet_copy_service.cc:433] P 1864f9ea44c54742b090c594a5be3669: ending tablet copy session facf1046c8054d2d841cf875c63e29e2-559610821917470fbcf5641d36acf409 on tablet 559610821917470fbcf5641d36acf409 with peer facf1046c8054d2d841cf875c63e29e2
I20240502 01:53:46.414728  1407 ts_tablet_manager.cc:902] T 86f2e43af7fb4e02bb853f0748d62ddb P 0aea20ce2a1144519ed144383c016d78: Initiating tablet copy from peer b191086bf5f2432b8d240510817ae5fe (127.25.109.129:42243)
I20240502 01:53:46.437976  1407 tablet_copy_client.cc:308] T 86f2e43af7fb4e02bb853f0748d62ddb P 0aea20ce2a1144519ed144383c016d78: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.129:42243
I20240502 01:53:46.703227 31671 tablet_copy_service.cc:140] P b191086bf5f2432b8d240510817ae5fe: Received BeginTabletCopySession request for tablet 86f2e43af7fb4e02bb853f0748d62ddb from peer 0aea20ce2a1144519ed144383c016d78 ({username='slave'} at 127.25.109.132:45639)
I20240502 01:53:46.703558 31671 tablet_copy_service.cc:161] P b191086bf5f2432b8d240510817ae5fe: Beginning new tablet copy session on tablet 86f2e43af7fb4e02bb853f0748d62ddb from peer 0aea20ce2a1144519ed144383c016d78 at {username='slave'} at 127.25.109.132:45639: session id = 0aea20ce2a1144519ed144383c016d78-86f2e43af7fb4e02bb853f0748d62ddb
I20240502 01:53:46.707382 31671 tablet_copy_source_session.cc:205] T 86f2e43af7fb4e02bb853f0748d62ddb P b191086bf5f2432b8d240510817ae5fe: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:46.711334   920 raft_consensus.cc:1203] T 559610821917470fbcf5641d36acf409 P facf1046c8054d2d841cf875c63e29e2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.70->[1.71-1.86]   Dedup: 1.74->[1.75-1.86]
I20240502 01:53:46.718618  1407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86f2e43af7fb4e02bb853f0748d62ddb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:46.727170  1407 tablet_copy_client.cc:783] T 86f2e43af7fb4e02bb853f0748d62ddb P 0aea20ce2a1144519ed144383c016d78: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:46.727483  1407 tablet_copy_client.cc:647] T 86f2e43af7fb4e02bb853f0748d62ddb P 0aea20ce2a1144519ed144383c016d78: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:46.735069 31783 consensus_queue.cc:237] T 704700e8872b4675b41d6c172731a5e0 P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 114, Committed index: 114, Last appended: 1.114, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 115 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } }
W20240502 01:53:46.738464 31719 consensus_peers.cc:485] T 704700e8872b4675b41d6c172731a5e0 P 35f356f20e274db7bf020f78f1bc9ae7 -> Peer 676092c80dd74040bb49a06db1ac209c (127.25.109.134:38633): Couldn't send request to peer 676092c80dd74040bb49a06db1ac209c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 704700e8872b4675b41d6c172731a5e0. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:46.747465 31916 raft_consensus.cc:1261] T 704700e8872b4675b41d6c172731a5e0 P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Refusing update from remote peer 35f356f20e274db7bf020f78f1bc9ae7: Log matching property violated. Preceding OpId in replica: term: 1 index: 114. Preceding OpId from leader: term: 1 index: 115. (index mismatch)
I20240502 01:53:46.749147  1393 consensus_queue.cc:1035] T 704700e8872b4675b41d6c172731a5e0 P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 115, Last known committed idx: 114, Time since last communication: 0.000s
I20240502 01:53:46.761197  1407 tablet_copy_client.cc:523] T 86f2e43af7fb4e02bb853f0748d62ddb P 0aea20ce2a1144519ed144383c016d78: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:46.765722  1407 tablet_bootstrap.cc:492] T 86f2e43af7fb4e02bb853f0748d62ddb P 0aea20ce2a1144519ed144383c016d78: Bootstrap starting.
I20240502 01:53:46.775921   995 raft_consensus.cc:2932] T 704700e8872b4675b41d6c172731a5e0 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 LEADER]: Committing config change with OpId 1.115: config changed from index -1 to 115, VOTER 676092c80dd74040bb49a06db1ac209c (127.25.109.134) added. New config: { opid_index: 115 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } } }
I20240502 01:53:46.782054 31915 raft_consensus.cc:2932] T 704700e8872b4675b41d6c172731a5e0 P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Committing config change with OpId 1.115: config changed from index -1 to 115, VOTER 676092c80dd74040bb49a06db1ac209c (127.25.109.134) added. New config: { opid_index: 115 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } } }
I20240502 01:53:46.803946 31500 catalog_manager.cc:5484] T 704700e8872b4675b41d6c172731a5e0 P 35f356f20e274db7bf020f78f1bc9ae7 reported cstate change: config changed from index -1 to 115, VOTER 676092c80dd74040bb49a06db1ac209c (127.25.109.134) added. New cstate: current_term: 1 leader_uuid: "35f356f20e274db7bf020f78f1bc9ae7" committed_config { opid_index: 115 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } } }
I20240502 01:53:47.123090  1407 tablet_bootstrap.cc:492] T 86f2e43af7fb4e02bb853f0748d62ddb P 0aea20ce2a1144519ed144383c016d78: Bootstrap replayed 1/1 log segments. Stats: ops{read=125 overwritten=0 applied=124 ignored=0} inserts{seen=796 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 01:53:47.124303  1407 tablet_bootstrap.cc:492] T 86f2e43af7fb4e02bb853f0748d62ddb P 0aea20ce2a1144519ed144383c016d78: Bootstrap complete.
I20240502 01:53:47.124728  1407 ts_tablet_manager.cc:1367] T 86f2e43af7fb4e02bb853f0748d62ddb P 0aea20ce2a1144519ed144383c016d78: Time spent bootstrapping tablet: real 0.359s	user 0.208s	sys 0.008s
I20240502 01:53:47.125911  1407 raft_consensus.cc:348] T 86f2e43af7fb4e02bb853f0748d62ddb P 0aea20ce2a1144519ed144383c016d78 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } }
I20240502 01:53:47.126751  1407 raft_consensus.cc:729] T 86f2e43af7fb4e02bb853f0748d62ddb P 0aea20ce2a1144519ed144383c016d78 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aea20ce2a1144519ed144383c016d78, State: Initialized, Role: FOLLOWER
I20240502 01:53:47.127113  1407 consensus_queue.cc:260] T 86f2e43af7fb4e02bb853f0748d62ddb P 0aea20ce2a1144519ed144383c016d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 124, Last appended: 1.125, Last appended by leader: 125, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } }
I20240502 01:53:47.138067  1407 ts_tablet_manager.cc:1398] T 86f2e43af7fb4e02bb853f0748d62ddb P 0aea20ce2a1144519ed144383c016d78: Time spent starting tablet: real 0.013s	user 0.002s	sys 0.003s
I20240502 01:53:47.142709 31671 tablet_copy_service.cc:342] P b191086bf5f2432b8d240510817ae5fe: Request end of tablet copy session 0aea20ce2a1144519ed144383c016d78-86f2e43af7fb4e02bb853f0748d62ddb received from {username='slave'} at 127.25.109.132:45639
I20240502 01:53:47.142939 31671 tablet_copy_service.cc:433] P b191086bf5f2432b8d240510817ae5fe: ending tablet copy session 0aea20ce2a1144519ed144383c016d78-86f2e43af7fb4e02bb853f0748d62ddb on tablet 86f2e43af7fb4e02bb853f0748d62ddb with peer 0aea20ce2a1144519ed144383c016d78
W20240502 01:53:47.311038 31852 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.135:37253, user_credentials={real_user=slave}} blocked reactor thread for 58147us
I20240502 01:53:47.314440  1445 ts_tablet_manager.cc:902] T 704700e8872b4675b41d6c172731a5e0 P 676092c80dd74040bb49a06db1ac209c: Initiating tablet copy from peer 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130:41639)
I20240502 01:53:47.427441 31784 consensus_queue.cc:237] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 148, Committed index: 148, Last appended: 2.148, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 149 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } }
I20240502 01:53:47.438158 31644 raft_consensus.cc:1261] T 10aa745ed48d46a3b0558d35dee6a807 P b191086bf5f2432b8d240510817ae5fe [term 2 FOLLOWER]: Refusing update from remote peer 35f356f20e274db7bf020f78f1bc9ae7: Log matching property violated. Preceding OpId in replica: term: 2 index: 148. Preceding OpId from leader: term: 2 index: 149. (index mismatch)
I20240502 01:53:47.440675   995 consensus_queue.cc:1035] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 149, Last known committed idx: 147, Time since last communication: 0.000s
I20240502 01:53:47.449965   517 raft_consensus.cc:1203] T 86f2e43af7fb4e02bb853f0748d62ddb P 0aea20ce2a1144519ed144383c016d78 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.112->[1.113-1.131]   Dedup: 1.125->[1.126-1.131]
I20240502 01:53:47.451311  1390 raft_consensus.cc:2932] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [term 2 LEADER]: Committing config change with OpId 2.149: config changed from index -1 to 149, VOTER 0aea20ce2a1144519ed144383c016d78 (127.25.109.132) added. New config: { opid_index: 149 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } }
I20240502 01:53:47.465305 31644 raft_consensus.cc:2932] T 10aa745ed48d46a3b0558d35dee6a807 P b191086bf5f2432b8d240510817ae5fe [term 2 FOLLOWER]: Committing config change with OpId 2.149: config changed from index -1 to 149, VOTER 0aea20ce2a1144519ed144383c016d78 (127.25.109.132) added. New config: { opid_index: 149 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } }
I20240502 01:53:47.466209 31501 catalog_manager.cc:5484] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 reported cstate change: config changed from index -1 to 149, VOTER 0aea20ce2a1144519ed144383c016d78 (127.25.109.132) added. New cstate: current_term: 2 leader_uuid: "35f356f20e274db7bf020f78f1bc9ae7" committed_config { opid_index: 149 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } }
W20240502 01:53:47.519177 31716 consensus_peers.cc:485] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 -> Peer 0aea20ce2a1144519ed144383c016d78 (127.25.109.132:34649): Couldn't send request to peer 0aea20ce2a1144519ed144383c016d78. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 10aa745ed48d46a3b0558d35dee6a807. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:47.677858  1445 tablet_copy_client.cc:308] T 704700e8872b4675b41d6c172731a5e0 P 676092c80dd74040bb49a06db1ac209c: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.130:41639
I20240502 01:53:47.678764 31805 tablet_copy_service.cc:140] P 35f356f20e274db7bf020f78f1bc9ae7: Received BeginTabletCopySession request for tablet 704700e8872b4675b41d6c172731a5e0 from peer 676092c80dd74040bb49a06db1ac209c ({username='slave'} at 127.25.109.134:45991)
I20240502 01:53:47.678936 31805 tablet_copy_service.cc:161] P 35f356f20e274db7bf020f78f1bc9ae7: Beginning new tablet copy session on tablet 704700e8872b4675b41d6c172731a5e0 from peer 676092c80dd74040bb49a06db1ac209c at {username='slave'} at 127.25.109.134:45991: session id = 676092c80dd74040bb49a06db1ac209c-704700e8872b4675b41d6c172731a5e0
I20240502 01:53:47.681751 31805 tablet_copy_source_session.cc:205] T 704700e8872b4675b41d6c172731a5e0 P 35f356f20e274db7bf020f78f1bc9ae7: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:47.683120  1445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 704700e8872b4675b41d6c172731a5e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:47.689436  1445 tablet_copy_client.cc:783] T 704700e8872b4675b41d6c172731a5e0 P 676092c80dd74040bb49a06db1ac209c: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:47.689754  1445 tablet_copy_client.cc:647] T 704700e8872b4675b41d6c172731a5e0 P 676092c80dd74040bb49a06db1ac209c: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:47.713076  1445 tablet_copy_client.cc:523] T 704700e8872b4675b41d6c172731a5e0 P 676092c80dd74040bb49a06db1ac209c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:47.723445  1445 tablet_bootstrap.cc:492] T 704700e8872b4675b41d6c172731a5e0 P 676092c80dd74040bb49a06db1ac209c: Bootstrap starting.
I20240502 01:53:48.041976  1474 ts_tablet_manager.cc:902] T 10aa745ed48d46a3b0558d35dee6a807 P 0aea20ce2a1144519ed144383c016d78: Initiating tablet copy from peer 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130:41639)
I20240502 01:53:48.055047  1445 tablet_bootstrap.cc:492] T 704700e8872b4675b41d6c172731a5e0 P 676092c80dd74040bb49a06db1ac209c: Bootstrap replayed 1/1 log segments. Stats: ops{read=121 overwritten=0 applied=121 ignored=0} inserts{seen=752 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:48.055723  1445 tablet_bootstrap.cc:492] T 704700e8872b4675b41d6c172731a5e0 P 676092c80dd74040bb49a06db1ac209c: Bootstrap complete.
I20240502 01:53:48.056164  1445 ts_tablet_manager.cc:1367] T 704700e8872b4675b41d6c172731a5e0 P 676092c80dd74040bb49a06db1ac209c: Time spent bootstrapping tablet: real 0.333s	user 0.183s	sys 0.021s
I20240502 01:53:48.057685  1445 raft_consensus.cc:348] T 704700e8872b4675b41d6c172731a5e0 P 676092c80dd74040bb49a06db1ac209c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 115 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } }
I20240502 01:53:48.058061  1445 raft_consensus.cc:729] T 704700e8872b4675b41d6c172731a5e0 P 676092c80dd74040bb49a06db1ac209c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 676092c80dd74040bb49a06db1ac209c, State: Initialized, Role: FOLLOWER
I20240502 01:53:48.061470  1445 consensus_queue.cc:260] T 704700e8872b4675b41d6c172731a5e0 P 676092c80dd74040bb49a06db1ac209c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 121, Last appended: 1.121, Last appended by leader: 121, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 115 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } }
I20240502 01:53:48.072759  1445 ts_tablet_manager.cc:1398] T 704700e8872b4675b41d6c172731a5e0 P 676092c80dd74040bb49a06db1ac209c: Time spent starting tablet: real 0.013s	user 0.005s	sys 0.002s
I20240502 01:53:48.076761 31805 tablet_copy_service.cc:342] P 35f356f20e274db7bf020f78f1bc9ae7: Request end of tablet copy session 676092c80dd74040bb49a06db1ac209c-704700e8872b4675b41d6c172731a5e0 received from {username='slave'} at 127.25.109.134:45991
I20240502 01:53:48.076992 31805 tablet_copy_service.cc:433] P 35f356f20e274db7bf020f78f1bc9ae7: ending tablet copy session 676092c80dd74040bb49a06db1ac209c-704700e8872b4675b41d6c172731a5e0 on tablet 704700e8872b4675b41d6c172731a5e0 with peer 676092c80dd74040bb49a06db1ac209c
I20240502 01:53:48.082520   786 raft_consensus.cc:1203] T 704700e8872b4675b41d6c172731a5e0 P 676092c80dd74040bb49a06db1ac209c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.114->[1.115-1.128]   Dedup: 1.121->[1.122-1.128]
I20240502 01:53:48.139752  1474 tablet_copy_client.cc:308] T 10aa745ed48d46a3b0558d35dee6a807 P 0aea20ce2a1144519ed144383c016d78: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.130:41639
I20240502 01:53:48.140643 31805 tablet_copy_service.cc:140] P 35f356f20e274db7bf020f78f1bc9ae7: Received BeginTabletCopySession request for tablet 10aa745ed48d46a3b0558d35dee6a807 from peer 0aea20ce2a1144519ed144383c016d78 ({username='slave'} at 127.25.109.132:56925)
I20240502 01:53:48.140853 31805 tablet_copy_service.cc:161] P 35f356f20e274db7bf020f78f1bc9ae7: Beginning new tablet copy session on tablet 10aa745ed48d46a3b0558d35dee6a807 from peer 0aea20ce2a1144519ed144383c016d78 at {username='slave'} at 127.25.109.132:56925: session id = 0aea20ce2a1144519ed144383c016d78-10aa745ed48d46a3b0558d35dee6a807
I20240502 01:53:48.144004 31805 tablet_copy_source_session.cc:205] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:48.145656  1474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10aa745ed48d46a3b0558d35dee6a807. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:48.160586  1474 tablet_copy_client.cc:783] T 10aa745ed48d46a3b0558d35dee6a807 P 0aea20ce2a1144519ed144383c016d78: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:48.161162  1474 tablet_copy_client.cc:647] T 10aa745ed48d46a3b0558d35dee6a807 P 0aea20ce2a1144519ed144383c016d78: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:48.188668  1474 tablet_copy_client.cc:523] T 10aa745ed48d46a3b0558d35dee6a807 P 0aea20ce2a1144519ed144383c016d78: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:48.194557  1474 tablet_bootstrap.cc:492] T 10aa745ed48d46a3b0558d35dee6a807 P 0aea20ce2a1144519ed144383c016d78: Bootstrap starting.
I20240502 01:53:48.443557 31644 consensus_queue.cc:237] T 02b02a474b5e4f6ab173ced2dc21c03e P b191086bf5f2432b8d240510817ae5fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 158, Committed index: 158, Last appended: 1.158, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } }
I20240502 01:53:48.447227 31783 raft_consensus.cc:1261] T 02b02a474b5e4f6ab173ced2dc21c03e P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 FOLLOWER]: Refusing update from remote peer b191086bf5f2432b8d240510817ae5fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 158. Preceding OpId from leader: term: 1 index: 159. (index mismatch)
I20240502 01:53:48.448287  1207 consensus_queue.cc:1035] T 02b02a474b5e4f6ab173ced2dc21c03e P b191086bf5f2432b8d240510817ae5fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 159, Last known committed idx: 158, Time since last communication: 0.000s
I20240502 01:53:48.452592  1207 raft_consensus.cc:2932] T 02b02a474b5e4f6ab173ced2dc21c03e P b191086bf5f2432b8d240510817ae5fe [term 1 LEADER]: Committing config change with OpId 1.159: config changed from index -1 to 159, VOTER facf1046c8054d2d841cf875c63e29e2 (127.25.109.135) added. New config: { opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } } }
I20240502 01:53:48.454758 31783 raft_consensus.cc:2932] T 02b02a474b5e4f6ab173ced2dc21c03e P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 FOLLOWER]: Committing config change with OpId 1.159: config changed from index -1 to 159, VOTER facf1046c8054d2d841cf875c63e29e2 (127.25.109.135) added. New config: { opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } } }
I20240502 01:53:48.460458 31499 catalog_manager.cc:5484] T 02b02a474b5e4f6ab173ced2dc21c03e P 35f356f20e274db7bf020f78f1bc9ae7 reported cstate change: config changed from index -1 to 159, VOTER facf1046c8054d2d841cf875c63e29e2 (127.25.109.135) added. New cstate: current_term: 1 leader_uuid: "b191086bf5f2432b8d240510817ae5fe" committed_config { opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } } }
W20240502 01:53:48.584396 31584 consensus_peers.cc:485] T 02b02a474b5e4f6ab173ced2dc21c03e P b191086bf5f2432b8d240510817ae5fe -> Peer facf1046c8054d2d841cf875c63e29e2 (127.25.109.135:37253): Couldn't send request to peer facf1046c8054d2d841cf875c63e29e2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 02b02a474b5e4f6ab173ced2dc21c03e. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:48.616906  1474 tablet_bootstrap.cc:492] T 10aa745ed48d46a3b0558d35dee6a807 P 0aea20ce2a1144519ed144383c016d78: Bootstrap replayed 1/1 log segments. Stats: ops{read=158 overwritten=0 applied=158 ignored=0} inserts{seen=789 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:48.622121  1474 tablet_bootstrap.cc:492] T 10aa745ed48d46a3b0558d35dee6a807 P 0aea20ce2a1144519ed144383c016d78: Bootstrap complete.
I20240502 01:53:48.622602  1474 ts_tablet_manager.cc:1367] T 10aa745ed48d46a3b0558d35dee6a807 P 0aea20ce2a1144519ed144383c016d78: Time spent bootstrapping tablet: real 0.428s	user 0.235s	sys 0.037s
I20240502 01:53:48.623961  1474 raft_consensus.cc:348] T 10aa745ed48d46a3b0558d35dee6a807 P 0aea20ce2a1144519ed144383c016d78 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 149 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } }
I20240502 01:53:48.624320  1474 raft_consensus.cc:729] T 10aa745ed48d46a3b0558d35dee6a807 P 0aea20ce2a1144519ed144383c016d78 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aea20ce2a1144519ed144383c016d78, State: Initialized, Role: FOLLOWER
I20240502 01:53:48.624686  1474 consensus_queue.cc:260] T 10aa745ed48d46a3b0558d35dee6a807 P 0aea20ce2a1144519ed144383c016d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 158, Last appended: 2.158, Last appended by leader: 158, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 149 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } }
I20240502 01:53:48.656390  1474 ts_tablet_manager.cc:1398] T 10aa745ed48d46a3b0558d35dee6a807 P 0aea20ce2a1144519ed144383c016d78: Time spent starting tablet: real 0.034s	user 0.013s	sys 0.006s
I20240502 01:53:48.660434 31805 tablet_copy_service.cc:342] P 35f356f20e274db7bf020f78f1bc9ae7: Request end of tablet copy session 0aea20ce2a1144519ed144383c016d78-10aa745ed48d46a3b0558d35dee6a807 received from {username='slave'} at 127.25.109.132:56925
I20240502 01:53:48.660605 31805 tablet_copy_service.cc:433] P 35f356f20e274db7bf020f78f1bc9ae7: ending tablet copy session 0aea20ce2a1144519ed144383c016d78-10aa745ed48d46a3b0558d35dee6a807 on tablet 10aa745ed48d46a3b0558d35dee6a807 with peer 0aea20ce2a1144519ed144383c016d78
I20240502 01:53:48.827937 31784 consensus_queue.cc:237] T fe2daac9229e44c9bee5f7fbaf886728 P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 158, Committed index: 158, Last appended: 1.158, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } }
I20240502 01:53:48.831584 31648 raft_consensus.cc:1261] T fe2daac9229e44c9bee5f7fbaf886728 P b191086bf5f2432b8d240510817ae5fe [term 1 FOLLOWER]: Refusing update from remote peer 35f356f20e274db7bf020f78f1bc9ae7: Log matching property violated. Preceding OpId in replica: term: 1 index: 158. Preceding OpId from leader: term: 1 index: 159. (index mismatch)
I20240502 01:53:48.835973  1390 consensus_queue.cc:1035] T fe2daac9229e44c9bee5f7fbaf886728 P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 159, Last known committed idx: 158, Time since last communication: 0.000s
I20240502 01:53:48.841293  1390 raft_consensus.cc:2932] T fe2daac9229e44c9bee5f7fbaf886728 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 LEADER]: Committing config change with OpId 1.159: config changed from index -1 to 159, VOTER 676092c80dd74040bb49a06db1ac209c (127.25.109.134) added. New config: { opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } } }
W20240502 01:53:48.842983 31719 consensus_peers.cc:485] T fe2daac9229e44c9bee5f7fbaf886728 P 35f356f20e274db7bf020f78f1bc9ae7 -> Peer 676092c80dd74040bb49a06db1ac209c (127.25.109.134:38633): Couldn't send request to peer 676092c80dd74040bb49a06db1ac209c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe2daac9229e44c9bee5f7fbaf886728. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:48.845077 31644 raft_consensus.cc:2932] T fe2daac9229e44c9bee5f7fbaf886728 P b191086bf5f2432b8d240510817ae5fe [term 1 FOLLOWER]: Committing config change with OpId 1.159: config changed from index -1 to 159, VOTER 676092c80dd74040bb49a06db1ac209c (127.25.109.134) added. New config: { opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } } }
I20240502 01:53:48.858284 31499 catalog_manager.cc:5484] T fe2daac9229e44c9bee5f7fbaf886728 P b191086bf5f2432b8d240510817ae5fe reported cstate change: config changed from index -1 to 159, VOTER 676092c80dd74040bb49a06db1ac209c (127.25.109.134) added. New cstate: current_term: 1 leader_uuid: "35f356f20e274db7bf020f78f1bc9ae7" committed_config { opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } } }
I20240502 01:53:48.954202  1522 ts_tablet_manager.cc:902] T 02b02a474b5e4f6ab173ced2dc21c03e P facf1046c8054d2d841cf875c63e29e2: Initiating tablet copy from peer b191086bf5f2432b8d240510817ae5fe (127.25.109.129:42243)
I20240502 01:53:49.002447  1522 tablet_copy_client.cc:308] T 02b02a474b5e4f6ab173ced2dc21c03e P facf1046c8054d2d841cf875c63e29e2: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.129:42243
I20240502 01:53:49.051143 31671 tablet_copy_service.cc:140] P b191086bf5f2432b8d240510817ae5fe: Received BeginTabletCopySession request for tablet 02b02a474b5e4f6ab173ced2dc21c03e from peer facf1046c8054d2d841cf875c63e29e2 ({username='slave'} at 127.25.109.135:39255)
I20240502 01:53:49.051359 31671 tablet_copy_service.cc:161] P b191086bf5f2432b8d240510817ae5fe: Beginning new tablet copy session on tablet 02b02a474b5e4f6ab173ced2dc21c03e from peer facf1046c8054d2d841cf875c63e29e2 at {username='slave'} at 127.25.109.135:39255: session id = facf1046c8054d2d841cf875c63e29e2-02b02a474b5e4f6ab173ced2dc21c03e
I20240502 01:53:49.054136 31671 tablet_copy_source_session.cc:205] T 02b02a474b5e4f6ab173ced2dc21c03e P b191086bf5f2432b8d240510817ae5fe: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:49.056205  1522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02b02a474b5e4f6ab173ced2dc21c03e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:49.067888  1522 tablet_copy_client.cc:783] T 02b02a474b5e4f6ab173ced2dc21c03e P facf1046c8054d2d841cf875c63e29e2: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:49.068969  1522 tablet_copy_client.cc:647] T 02b02a474b5e4f6ab173ced2dc21c03e P facf1046c8054d2d841cf875c63e29e2: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:49.071508  1522 tablet_copy_client.cc:523] T 02b02a474b5e4f6ab173ced2dc21c03e P facf1046c8054d2d841cf875c63e29e2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:49.084668  1522 tablet_bootstrap.cc:492] T 02b02a474b5e4f6ab173ced2dc21c03e P facf1046c8054d2d841cf875c63e29e2: Bootstrap starting.
I20240502 01:53:49.180117   517 raft_consensus.cc:1203] T 10aa745ed48d46a3b0558d35dee6a807 P 0aea20ce2a1144519ed144383c016d78 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.148->[2.149-2.175]   Dedup: 2.158->[2.159-2.175]
I20240502 01:53:49.306727  1545 ts_tablet_manager.cc:902] T fe2daac9229e44c9bee5f7fbaf886728 P 676092c80dd74040bb49a06db1ac209c: Initiating tablet copy from peer 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130:41639)
I20240502 01:53:49.353190 31785 consensus_queue.cc:237] T e714cba54201433caa218073097c1e6a P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 134, Committed index: 134, Last appended: 1.134, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 135 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } }
I20240502 01:53:49.426960 31916 raft_consensus.cc:1261] T e714cba54201433caa218073097c1e6a P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Refusing update from remote peer 35f356f20e274db7bf020f78f1bc9ae7: Log matching property violated. Preceding OpId in replica: term: 1 index: 134. Preceding OpId from leader: term: 1 index: 135. (index mismatch)
I20240502 01:53:49.435933   995 consensus_queue.cc:1035] T e714cba54201433caa218073097c1e6a P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 135, Last known committed idx: 134, Time since last communication: 0.000s
I20240502 01:53:49.479015  1393 raft_consensus.cc:2932] T e714cba54201433caa218073097c1e6a P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 LEADER]: Committing config change with OpId 1.135: config changed from index -1 to 135, VOTER facf1046c8054d2d841cf875c63e29e2 (127.25.109.135) added. New config: { opid_index: 135 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } } }
I20240502 01:53:49.485798 31499 catalog_manager.cc:5484] T e714cba54201433caa218073097c1e6a P 35f356f20e274db7bf020f78f1bc9ae7 reported cstate change: config changed from index -1 to 135, VOTER facf1046c8054d2d841cf875c63e29e2 (127.25.109.135) added. New cstate: current_term: 1 leader_uuid: "35f356f20e274db7bf020f78f1bc9ae7" committed_config { opid_index: 135 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } } }
I20240502 01:53:49.510658 31916 raft_consensus.cc:2932] T e714cba54201433caa218073097c1e6a P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Committing config change with OpId 1.135: config changed from index -1 to 135, VOTER facf1046c8054d2d841cf875c63e29e2 (127.25.109.135) added. New config: { opid_index: 135 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } } }
I20240502 01:53:49.756305  1522 tablet_bootstrap.cc:492] T 02b02a474b5e4f6ab173ced2dc21c03e P facf1046c8054d2d841cf875c63e29e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=176 overwritten=0 applied=176 ignored=0} inserts{seen=861 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:49.756970  1522 tablet_bootstrap.cc:492] T 02b02a474b5e4f6ab173ced2dc21c03e P facf1046c8054d2d841cf875c63e29e2: Bootstrap complete.
I20240502 01:53:49.757390  1522 ts_tablet_manager.cc:1367] T 02b02a474b5e4f6ab173ced2dc21c03e P facf1046c8054d2d841cf875c63e29e2: Time spent bootstrapping tablet: real 0.673s	user 0.292s	sys 0.061s
I20240502 01:53:49.758286  1522 raft_consensus.cc:348] T 02b02a474b5e4f6ab173ced2dc21c03e P facf1046c8054d2d841cf875c63e29e2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } }
I20240502 01:53:49.758669  1522 raft_consensus.cc:729] T 02b02a474b5e4f6ab173ced2dc21c03e P facf1046c8054d2d841cf875c63e29e2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: facf1046c8054d2d841cf875c63e29e2, State: Initialized, Role: FOLLOWER
I20240502 01:53:49.758988  1522 consensus_queue.cc:260] T 02b02a474b5e4f6ab173ced2dc21c03e P facf1046c8054d2d841cf875c63e29e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 176, Last appended: 1.176, Last appended by leader: 176, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } }
I20240502 01:53:49.767256  1522 ts_tablet_manager.cc:1398] T 02b02a474b5e4f6ab173ced2dc21c03e P facf1046c8054d2d841cf875c63e29e2: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.003s
I20240502 01:53:49.768705 31671 tablet_copy_service.cc:342] P b191086bf5f2432b8d240510817ae5fe: Request end of tablet copy session facf1046c8054d2d841cf875c63e29e2-02b02a474b5e4f6ab173ced2dc21c03e received from {username='slave'} at 127.25.109.135:39255
I20240502 01:53:49.768854 31671 tablet_copy_service.cc:433] P b191086bf5f2432b8d240510817ae5fe: ending tablet copy session facf1046c8054d2d841cf875c63e29e2-02b02a474b5e4f6ab173ced2dc21c03e on tablet 02b02a474b5e4f6ab173ced2dc21c03e with peer facf1046c8054d2d841cf875c63e29e2
W20240502 01:53:49.835712 31719 consensus_peers.cc:485] T e714cba54201433caa218073097c1e6a P 35f356f20e274db7bf020f78f1bc9ae7 -> Peer facf1046c8054d2d841cf875c63e29e2 (127.25.109.135:37253): Couldn't send request to peer facf1046c8054d2d841cf875c63e29e2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e714cba54201433caa218073097c1e6a. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:49.870777  1545 tablet_copy_client.cc:308] T fe2daac9229e44c9bee5f7fbaf886728 P 676092c80dd74040bb49a06db1ac209c: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.130:41639
I20240502 01:53:49.872498 31805 tablet_copy_service.cc:140] P 35f356f20e274db7bf020f78f1bc9ae7: Received BeginTabletCopySession request for tablet fe2daac9229e44c9bee5f7fbaf886728 from peer 676092c80dd74040bb49a06db1ac209c ({username='slave'} at 127.25.109.134:45991)
I20240502 01:53:49.872681 31805 tablet_copy_service.cc:161] P 35f356f20e274db7bf020f78f1bc9ae7: Beginning new tablet copy session on tablet fe2daac9229e44c9bee5f7fbaf886728 from peer 676092c80dd74040bb49a06db1ac209c at {username='slave'} at 127.25.109.134:45991: session id = 676092c80dd74040bb49a06db1ac209c-fe2daac9229e44c9bee5f7fbaf886728
I20240502 01:53:49.875560 31805 tablet_copy_source_session.cc:205] T fe2daac9229e44c9bee5f7fbaf886728 P 35f356f20e274db7bf020f78f1bc9ae7: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:49.878090  1545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe2daac9229e44c9bee5f7fbaf886728. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:49.900638  1545 tablet_copy_client.cc:783] T fe2daac9229e44c9bee5f7fbaf886728 P 676092c80dd74040bb49a06db1ac209c: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:49.901015  1545 tablet_copy_client.cc:647] T fe2daac9229e44c9bee5f7fbaf886728 P 676092c80dd74040bb49a06db1ac209c: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:49.914973  1545 tablet_copy_client.cc:523] T fe2daac9229e44c9bee5f7fbaf886728 P 676092c80dd74040bb49a06db1ac209c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:49.935596  1545 tablet_bootstrap.cc:492] T fe2daac9229e44c9bee5f7fbaf886728 P 676092c80dd74040bb49a06db1ac209c: Bootstrap starting.
I20240502 01:53:50.016136   919 raft_consensus.cc:1203] T 02b02a474b5e4f6ab173ced2dc21c03e P facf1046c8054d2d841cf875c63e29e2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.158->[1.159-1.189]   Dedup: 1.176->[1.177-1.189]
I20240502 01:53:50.258630  1522 ts_tablet_manager.cc:902] T e714cba54201433caa218073097c1e6a P facf1046c8054d2d841cf875c63e29e2: Initiating tablet copy from peer 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130:41639)
I20240502 01:53:50.299333  1522 tablet_copy_client.cc:308] T e714cba54201433caa218073097c1e6a P facf1046c8054d2d841cf875c63e29e2: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.130:41639
I20240502 01:53:50.607205  1545 tablet_bootstrap.cc:492] T fe2daac9229e44c9bee5f7fbaf886728 P 676092c80dd74040bb49a06db1ac209c: Bootstrap replayed 1/1 log segments. Stats: ops{read=175 overwritten=0 applied=175 ignored=0} inserts{seen=847 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:50.607848  1545 tablet_bootstrap.cc:492] T fe2daac9229e44c9bee5f7fbaf886728 P 676092c80dd74040bb49a06db1ac209c: Bootstrap complete.
I20240502 01:53:50.609611  1545 ts_tablet_manager.cc:1367] T fe2daac9229e44c9bee5f7fbaf886728 P 676092c80dd74040bb49a06db1ac209c: Time spent bootstrapping tablet: real 0.673s	user 0.252s	sys 0.020s
I20240502 01:53:50.611070  1545 raft_consensus.cc:348] T fe2daac9229e44c9bee5f7fbaf886728 P 676092c80dd74040bb49a06db1ac209c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } }
I20240502 01:53:50.611392  1545 raft_consensus.cc:729] T fe2daac9229e44c9bee5f7fbaf886728 P 676092c80dd74040bb49a06db1ac209c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 676092c80dd74040bb49a06db1ac209c, State: Initialized, Role: FOLLOWER
I20240502 01:53:50.612138  1545 consensus_queue.cc:260] T fe2daac9229e44c9bee5f7fbaf886728 P 676092c80dd74040bb49a06db1ac209c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 175, Last appended: 1.175, Last appended by leader: 175, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } }
I20240502 01:53:50.613008  1545 ts_tablet_manager.cc:1398] T fe2daac9229e44c9bee5f7fbaf886728 P 676092c80dd74040bb49a06db1ac209c: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:53:50.621383 31805 tablet_copy_service.cc:342] P 35f356f20e274db7bf020f78f1bc9ae7: Request end of tablet copy session 676092c80dd74040bb49a06db1ac209c-fe2daac9229e44c9bee5f7fbaf886728 received from {username='slave'} at 127.25.109.134:45991
I20240502 01:53:50.621577 31805 tablet_copy_service.cc:433] P 35f356f20e274db7bf020f78f1bc9ae7: ending tablet copy session 676092c80dd74040bb49a06db1ac209c-fe2daac9229e44c9bee5f7fbaf886728 on tablet fe2daac9229e44c9bee5f7fbaf886728 with peer 676092c80dd74040bb49a06db1ac209c
I20240502 01:53:50.635182 31805 tablet_copy_service.cc:140] P 35f356f20e274db7bf020f78f1bc9ae7: Received BeginTabletCopySession request for tablet e714cba54201433caa218073097c1e6a from peer facf1046c8054d2d841cf875c63e29e2 ({username='slave'} at 127.25.109.135:54601)
I20240502 01:53:50.635411 31805 tablet_copy_service.cc:161] P 35f356f20e274db7bf020f78f1bc9ae7: Beginning new tablet copy session on tablet e714cba54201433caa218073097c1e6a from peer facf1046c8054d2d841cf875c63e29e2 at {username='slave'} at 127.25.109.135:54601: session id = facf1046c8054d2d841cf875c63e29e2-e714cba54201433caa218073097c1e6a
I20240502 01:53:50.638790 31805 tablet_copy_source_session.cc:205] T e714cba54201433caa218073097c1e6a P 35f356f20e274db7bf020f78f1bc9ae7: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:50.640368  1522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e714cba54201433caa218073097c1e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:50.649873  1522 tablet_copy_client.cc:783] T e714cba54201433caa218073097c1e6a P facf1046c8054d2d841cf875c63e29e2: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:50.650254  1522 tablet_copy_client.cc:647] T e714cba54201433caa218073097c1e6a P facf1046c8054d2d841cf875c63e29e2: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:50.653249  1522 tablet_copy_client.cc:523] T e714cba54201433caa218073097c1e6a P facf1046c8054d2d841cf875c63e29e2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:50.656759  1522 tablet_bootstrap.cc:492] T e714cba54201433caa218073097c1e6a P facf1046c8054d2d841cf875c63e29e2: Bootstrap starting.
I20240502 01:53:50.681556 31785 consensus_queue.cc:237] T 82e5853a509c43df917f257186844ddc P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 141, Committed index: 141, Last appended: 1.141, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 142 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } }
W20240502 01:53:50.705807 31716 consensus_peers.cc:485] T 82e5853a509c43df917f257186844ddc P 35f356f20e274db7bf020f78f1bc9ae7 -> Peer 0aea20ce2a1144519ed144383c016d78 (127.25.109.132:34649): Couldn't send request to peer 0aea20ce2a1144519ed144383c016d78. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 82e5853a509c43df917f257186844ddc. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:50.707620 31915 raft_consensus.cc:1261] T 82e5853a509c43df917f257186844ddc P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Refusing update from remote peer 35f356f20e274db7bf020f78f1bc9ae7: Log matching property violated. Preceding OpId in replica: term: 1 index: 141. Preceding OpId from leader: term: 1 index: 142. (index mismatch)
I20240502 01:53:50.712286  1393 consensus_queue.cc:1035] T 82e5853a509c43df917f257186844ddc P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 142, Last known committed idx: 141, Time since last communication: 0.000s
I20240502 01:53:50.756078  1393 raft_consensus.cc:2932] T 82e5853a509c43df917f257186844ddc P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 LEADER]: Committing config change with OpId 1.142: config changed from index -1 to 142, VOTER 0aea20ce2a1144519ed144383c016d78 (127.25.109.132) added. New config: { opid_index: 142 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } }
I20240502 01:53:50.784530 31915 raft_consensus.cc:2932] T 82e5853a509c43df917f257186844ddc P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Committing config change with OpId 1.142: config changed from index -1 to 142, VOTER 0aea20ce2a1144519ed144383c016d78 (127.25.109.132) added. New config: { opid_index: 142 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } }
I20240502 01:53:50.792112 31499 catalog_manager.cc:5484] T 82e5853a509c43df917f257186844ddc P 1864f9ea44c54742b090c594a5be3669 reported cstate change: config changed from index -1 to 142, VOTER 0aea20ce2a1144519ed144383c016d78 (127.25.109.132) added. New cstate: current_term: 1 leader_uuid: "35f356f20e274db7bf020f78f1bc9ae7" committed_config { opid_index: 142 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } }
I20240502 01:53:50.805802   786 raft_consensus.cc:1203] T fe2daac9229e44c9bee5f7fbaf886728 P 676092c80dd74040bb49a06db1ac209c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.158->[1.159-1.182]   Dedup: 1.175->[1.176-1.182]
W20240502 01:53:50.971601 31719 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.134:38633, user_credentials={real_user=slave}} blocked reactor thread for 79910.3us
I20240502 01:53:51.433241  1629 ts_tablet_manager.cc:902] T 82e5853a509c43df917f257186844ddc P 0aea20ce2a1144519ed144383c016d78: Initiating tablet copy from peer 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130:41639)
I20240502 01:53:51.518222  1629 tablet_copy_client.cc:308] T 82e5853a509c43df917f257186844ddc P 0aea20ce2a1144519ed144383c016d78: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.130:41639
I20240502 01:53:51.519222 31805 tablet_copy_service.cc:140] P 35f356f20e274db7bf020f78f1bc9ae7: Received BeginTabletCopySession request for tablet 82e5853a509c43df917f257186844ddc from peer 0aea20ce2a1144519ed144383c016d78 ({username='slave'} at 127.25.109.132:56925)
I20240502 01:53:51.519420 31805 tablet_copy_service.cc:161] P 35f356f20e274db7bf020f78f1bc9ae7: Beginning new tablet copy session on tablet 82e5853a509c43df917f257186844ddc from peer 0aea20ce2a1144519ed144383c016d78 at {username='slave'} at 127.25.109.132:56925: session id = 0aea20ce2a1144519ed144383c016d78-82e5853a509c43df917f257186844ddc
I20240502 01:53:51.522380 31805 tablet_copy_source_session.cc:205] T 82e5853a509c43df917f257186844ddc P 35f356f20e274db7bf020f78f1bc9ae7: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:51.528157  1629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82e5853a509c43df917f257186844ddc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:51.566550  1629 tablet_copy_client.cc:783] T 82e5853a509c43df917f257186844ddc P 0aea20ce2a1144519ed144383c016d78: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:51.574355  1629 tablet_copy_client.cc:647] T 82e5853a509c43df917f257186844ddc P 0aea20ce2a1144519ed144383c016d78: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:51.579303  1522 tablet_bootstrap.cc:492] T e714cba54201433caa218073097c1e6a P facf1046c8054d2d841cf875c63e29e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=146 overwritten=0 applied=146 ignored=0} inserts{seen=777 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:51.580022  1629 tablet_copy_client.cc:523] T 82e5853a509c43df917f257186844ddc P 0aea20ce2a1144519ed144383c016d78: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:51.584112  1629 tablet_bootstrap.cc:492] T 82e5853a509c43df917f257186844ddc P 0aea20ce2a1144519ed144383c016d78: Bootstrap starting.
I20240502 01:53:51.592895  1522 tablet_bootstrap.cc:492] T e714cba54201433caa218073097c1e6a P facf1046c8054d2d841cf875c63e29e2: Bootstrap complete.
I20240502 01:53:51.594933  1522 ts_tablet_manager.cc:1367] T e714cba54201433caa218073097c1e6a P facf1046c8054d2d841cf875c63e29e2: Time spent bootstrapping tablet: real 0.938s	user 0.216s	sys 0.014s
I20240502 01:53:51.595984  1522 raft_consensus.cc:348] T e714cba54201433caa218073097c1e6a P facf1046c8054d2d841cf875c63e29e2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 135 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } }
I20240502 01:53:51.596321  1522 raft_consensus.cc:729] T e714cba54201433caa218073097c1e6a P facf1046c8054d2d841cf875c63e29e2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: facf1046c8054d2d841cf875c63e29e2, State: Initialized, Role: FOLLOWER
I20240502 01:53:51.596704  1522 consensus_queue.cc:260] T e714cba54201433caa218073097c1e6a P facf1046c8054d2d841cf875c63e29e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 146, Last appended: 1.146, Last appended by leader: 146, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 135 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } }
I20240502 01:53:51.616101  1522 ts_tablet_manager.cc:1398] T e714cba54201433caa218073097c1e6a P facf1046c8054d2d841cf875c63e29e2: Time spent starting tablet: real 0.021s	user 0.001s	sys 0.004s
I20240502 01:53:51.616968 31805 tablet_copy_service.cc:342] P 35f356f20e274db7bf020f78f1bc9ae7: Request end of tablet copy session facf1046c8054d2d841cf875c63e29e2-e714cba54201433caa218073097c1e6a received from {username='slave'} at 127.25.109.135:54601
I20240502 01:53:51.617151 31805 tablet_copy_service.cc:433] P 35f356f20e274db7bf020f78f1bc9ae7: ending tablet copy session facf1046c8054d2d841cf875c63e29e2-e714cba54201433caa218073097c1e6a on tablet e714cba54201433caa218073097c1e6a with peer facf1046c8054d2d841cf875c63e29e2
I20240502 01:53:51.683609 31784 consensus_queue.cc:237] T 6b56adc64eba4898b80687a2679b70c8 P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 190, Committed index: 190, Last appended: 1.191, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 192 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } }
I20240502 01:53:51.686327 31647 raft_consensus.cc:1261] T 6b56adc64eba4898b80687a2679b70c8 P b191086bf5f2432b8d240510817ae5fe [term 1 FOLLOWER]: Refusing update from remote peer 35f356f20e274db7bf020f78f1bc9ae7: Log matching property violated. Preceding OpId in replica: term: 1 index: 190. Preceding OpId from leader: term: 1 index: 192. (index mismatch)
I20240502 01:53:51.687674  1390 consensus_queue.cc:1035] T 6b56adc64eba4898b80687a2679b70c8 P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 192, Last known committed idx: 190, Time since last communication: 0.000s
I20240502 01:53:51.690017 31645 raft_consensus.cc:1203] T 6b56adc64eba4898b80687a2679b70c8 P b191086bf5f2432b8d240510817ae5fe [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.190->[1.191-1.192]   Dedup: 1.191->[1.192-1.192]
I20240502 01:53:51.694023  1616 raft_consensus.cc:2932] T 6b56adc64eba4898b80687a2679b70c8 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 LEADER]: Committing config change with OpId 1.192: config changed from index -1 to 192, VOTER 676092c80dd74040bb49a06db1ac209c (127.25.109.134) added. New config: { opid_index: 192 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } } }
I20240502 01:53:51.703081 31502 catalog_manager.cc:5484] T 6b56adc64eba4898b80687a2679b70c8 P 35f356f20e274db7bf020f78f1bc9ae7 reported cstate change: config changed from index -1 to 192, VOTER 676092c80dd74040bb49a06db1ac209c (127.25.109.134) added. New cstate: current_term: 1 leader_uuid: "35f356f20e274db7bf020f78f1bc9ae7" committed_config { opid_index: 192 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } } }
W20240502 01:53:51.705476 31719 consensus_peers.cc:485] T 6b56adc64eba4898b80687a2679b70c8 P 35f356f20e274db7bf020f78f1bc9ae7 -> Peer 676092c80dd74040bb49a06db1ac209c (127.25.109.134:38633): Couldn't send request to peer 676092c80dd74040bb49a06db1ac209c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b56adc64eba4898b80687a2679b70c8. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:51.709421 31648 raft_consensus.cc:2932] T 6b56adc64eba4898b80687a2679b70c8 P b191086bf5f2432b8d240510817ae5fe [term 1 FOLLOWER]: Committing config change with OpId 1.192: config changed from index -1 to 192, VOTER 676092c80dd74040bb49a06db1ac209c (127.25.109.134) added. New config: { opid_index: 192 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } } }
I20240502 01:53:51.762116   920 raft_consensus.cc:1203] T e714cba54201433caa218073097c1e6a P facf1046c8054d2d841cf875c63e29e2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.134->[1.135-1.163]   Dedup: 1.146->[1.147-1.163]
I20240502 01:53:52.111994  1673 ts_tablet_manager.cc:902] T 6b56adc64eba4898b80687a2679b70c8 P 676092c80dd74040bb49a06db1ac209c: Initiating tablet copy from peer 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130:41639)
I20240502 01:53:52.254575  1673 tablet_copy_client.cc:308] T 6b56adc64eba4898b80687a2679b70c8 P 676092c80dd74040bb49a06db1ac209c: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.130:41639
I20240502 01:53:52.255750 31805 tablet_copy_service.cc:140] P 35f356f20e274db7bf020f78f1bc9ae7: Received BeginTabletCopySession request for tablet 6b56adc64eba4898b80687a2679b70c8 from peer 676092c80dd74040bb49a06db1ac209c ({username='slave'} at 127.25.109.134:45991)
I20240502 01:53:52.255996 31805 tablet_copy_service.cc:161] P 35f356f20e274db7bf020f78f1bc9ae7: Beginning new tablet copy session on tablet 6b56adc64eba4898b80687a2679b70c8 from peer 676092c80dd74040bb49a06db1ac209c at {username='slave'} at 127.25.109.134:45991: session id = 676092c80dd74040bb49a06db1ac209c-6b56adc64eba4898b80687a2679b70c8
I20240502 01:53:52.258919 31805 tablet_copy_source_session.cc:205] T 6b56adc64eba4898b80687a2679b70c8 P 35f356f20e274db7bf020f78f1bc9ae7: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:52.260373  1673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b56adc64eba4898b80687a2679b70c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:52.362283  1673 tablet_copy_client.cc:783] T 6b56adc64eba4898b80687a2679b70c8 P 676092c80dd74040bb49a06db1ac209c: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:52.364329  1673 tablet_copy_client.cc:647] T 6b56adc64eba4898b80687a2679b70c8 P 676092c80dd74040bb49a06db1ac209c: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:52.369652  1673 tablet_copy_client.cc:523] T 6b56adc64eba4898b80687a2679b70c8 P 676092c80dd74040bb49a06db1ac209c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:52.385275  1673 tablet_bootstrap.cc:492] T 6b56adc64eba4898b80687a2679b70c8 P 676092c80dd74040bb49a06db1ac209c: Bootstrap starting.
I20240502 01:53:52.499667  1629 tablet_bootstrap.cc:492] T 82e5853a509c43df917f257186844ddc P 0aea20ce2a1144519ed144383c016d78: Bootstrap replayed 1/1 log segments. Stats: ops{read=157 overwritten=0 applied=156 ignored=0} inserts{seen=841 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 01:53:52.500275  1629 tablet_bootstrap.cc:492] T 82e5853a509c43df917f257186844ddc P 0aea20ce2a1144519ed144383c016d78: Bootstrap complete.
I20240502 01:53:52.500690  1629 ts_tablet_manager.cc:1367] T 82e5853a509c43df917f257186844ddc P 0aea20ce2a1144519ed144383c016d78: Time spent bootstrapping tablet: real 0.917s	user 0.231s	sys 0.028s
I20240502 01:53:52.501645  1629 raft_consensus.cc:348] T 82e5853a509c43df917f257186844ddc P 0aea20ce2a1144519ed144383c016d78 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 142 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } }
I20240502 01:53:52.505481  1629 raft_consensus.cc:729] T 82e5853a509c43df917f257186844ddc P 0aea20ce2a1144519ed144383c016d78 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aea20ce2a1144519ed144383c016d78, State: Initialized, Role: FOLLOWER
I20240502 01:53:52.507911  1629 consensus_queue.cc:260] T 82e5853a509c43df917f257186844ddc P 0aea20ce2a1144519ed144383c016d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 156, Last appended: 1.157, Last appended by leader: 157, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 142 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } }
I20240502 01:53:52.522107  1629 ts_tablet_manager.cc:1398] T 82e5853a509c43df917f257186844ddc P 0aea20ce2a1144519ed144383c016d78: Time spent starting tablet: real 0.021s	user 0.004s	sys 0.000s
I20240502 01:53:52.524140 31805 tablet_copy_service.cc:342] P 35f356f20e274db7bf020f78f1bc9ae7: Request end of tablet copy session 0aea20ce2a1144519ed144383c016d78-82e5853a509c43df917f257186844ddc received from {username='slave'} at 127.25.109.132:56925
I20240502 01:53:52.524289 31805 tablet_copy_service.cc:433] P 35f356f20e274db7bf020f78f1bc9ae7: ending tablet copy session 0aea20ce2a1144519ed144383c016d78-82e5853a509c43df917f257186844ddc on tablet 82e5853a509c43df917f257186844ddc with peer 0aea20ce2a1144519ed144383c016d78
I20240502 01:53:52.706809   517 raft_consensus.cc:1203] T 82e5853a509c43df917f257186844ddc P 0aea20ce2a1144519ed144383c016d78 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.141->[1.142-1.174]   Dedup: 1.157->[1.158-1.174]
I20240502 01:53:53.057680  1673 tablet_bootstrap.cc:492] T 6b56adc64eba4898b80687a2679b70c8 P 676092c80dd74040bb49a06db1ac209c: Bootstrap replayed 1/1 log segments. Stats: ops{read=200 overwritten=0 applied=200 ignored=0} inserts{seen=872 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:53.058389  1673 tablet_bootstrap.cc:492] T 6b56adc64eba4898b80687a2679b70c8 P 676092c80dd74040bb49a06db1ac209c: Bootstrap complete.
I20240502 01:53:53.058802  1673 ts_tablet_manager.cc:1367] T 6b56adc64eba4898b80687a2679b70c8 P 676092c80dd74040bb49a06db1ac209c: Time spent bootstrapping tablet: real 0.674s	user 0.267s	sys 0.024s
I20240502 01:53:53.059909  1673 raft_consensus.cc:348] T 6b56adc64eba4898b80687a2679b70c8 P 676092c80dd74040bb49a06db1ac209c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 192 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } }
I20240502 01:53:53.060326  1673 raft_consensus.cc:729] T 6b56adc64eba4898b80687a2679b70c8 P 676092c80dd74040bb49a06db1ac209c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 676092c80dd74040bb49a06db1ac209c, State: Initialized, Role: FOLLOWER
I20240502 01:53:53.060844  1673 consensus_queue.cc:260] T 6b56adc64eba4898b80687a2679b70c8 P 676092c80dd74040bb49a06db1ac209c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 200, Last appended: 1.200, Last appended by leader: 200, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 192 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } }
I20240502 01:53:53.068785  1673 ts_tablet_manager.cc:1398] T 6b56adc64eba4898b80687a2679b70c8 P 676092c80dd74040bb49a06db1ac209c: Time spent starting tablet: real 0.010s	user 0.004s	sys 0.004s
W20240502 01:53:53.074463 31719 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.134:38633, user_credentials={real_user=slave}} blocked reactor thread for 136759us
I20240502 01:53:53.076064 31805 tablet_copy_service.cc:342] P 35f356f20e274db7bf020f78f1bc9ae7: Request end of tablet copy session 676092c80dd74040bb49a06db1ac209c-6b56adc64eba4898b80687a2679b70c8 received from {username='slave'} at 127.25.109.134:45991
I20240502 01:53:53.076218 31805 tablet_copy_service.cc:433] P 35f356f20e274db7bf020f78f1bc9ae7: ending tablet copy session 676092c80dd74040bb49a06db1ac209c-6b56adc64eba4898b80687a2679b70c8 on tablet 6b56adc64eba4898b80687a2679b70c8 with peer 676092c80dd74040bb49a06db1ac209c
I20240502 01:53:53.108768   785 raft_consensus.cc:1203] T 6b56adc64eba4898b80687a2679b70c8 P 676092c80dd74040bb49a06db1ac209c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.191->[1.192-1.212]   Dedup: 1.200->[1.201-1.212]
I20240502 01:53:53.240120   497 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:53.267267 31896 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:53:53.335505 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:53.386317   630 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:53.411463   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:53.414294   766 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:53.471060 31764 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 01:53:54.152456 31581 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.132:34649, user_credentials={real_user=slave}} blocked reactor thread for 44517.6us
I20240502 01:53:54.285984 31785 consensus_queue.cc:237] T 6b56adc64eba4898b80687a2679b70c8 P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 228, Committed index: 228, Last appended: 1.228, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } }
I20240502 01:53:54.297564   784 raft_consensus.cc:1261] T 6b56adc64eba4898b80687a2679b70c8 P 676092c80dd74040bb49a06db1ac209c [term 1 FOLLOWER]: Refusing update from remote peer 35f356f20e274db7bf020f78f1bc9ae7: Log matching property violated. Preceding OpId in replica: term: 1 index: 228. Preceding OpId from leader: term: 1 index: 229. (index mismatch)
I20240502 01:53:54.299669  1605 consensus_queue.cc:1035] T 6b56adc64eba4898b80687a2679b70c8 P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 229, Last known committed idx: 228, Time since last communication: 0.000s
I20240502 01:53:54.313341  1604 raft_consensus.cc:2932] T 6b56adc64eba4898b80687a2679b70c8 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 LEADER]: Committing config change with OpId 1.229: config changed from index 192 to 229, VOTER b191086bf5f2432b8d240510817ae5fe (127.25.109.129) evicted. New config: { opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } } }
I20240502 01:53:54.315814   784 raft_consensus.cc:2932] T 6b56adc64eba4898b80687a2679b70c8 P 676092c80dd74040bb49a06db1ac209c [term 1 FOLLOWER]: Committing config change with OpId 1.229: config changed from index 192 to 229, VOTER b191086bf5f2432b8d240510817ae5fe (127.25.109.129) evicted. New config: { opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } } }
I20240502 01:53:54.324334 31502 catalog_manager.cc:5484] T 6b56adc64eba4898b80687a2679b70c8 P 35f356f20e274db7bf020f78f1bc9ae7 reported cstate change: config changed from index 192 to 229, VOTER b191086bf5f2432b8d240510817ae5fe (127.25.109.129) evicted. New cstate: current_term: 1 leader_uuid: "35f356f20e274db7bf020f78f1bc9ae7" committed_config { opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } } }
I20240502 01:53:54.517751 31628 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b56adc64eba4898b80687a2679b70c8 with delete_type TABLET_DATA_TOMBSTONED (TS b191086bf5f2432b8d240510817ae5fe not found in new config with opid_index 229) from {username='slave'} at 127.0.0.1:48324
I20240502 01:53:54.532274  1775 tablet_replica.cc:330] T 6b56adc64eba4898b80687a2679b70c8 P b191086bf5f2432b8d240510817ae5fe: stopping tablet replica
I20240502 01:53:54.532966  1775 raft_consensus.cc:2229] T 6b56adc64eba4898b80687a2679b70c8 P b191086bf5f2432b8d240510817ae5fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:54.586524  1775 pending_rounds.cc:62] T 6b56adc64eba4898b80687a2679b70c8 P b191086bf5f2432b8d240510817ae5fe: Trying to abort 1 pending ops.
I20240502 01:53:54.588580  1775 pending_rounds.cc:68] T 6b56adc64eba4898b80687a2679b70c8 P b191086bf5f2432b8d240510817ae5fe: Aborting op as it isn't in flight: id { term: 1 index: 228 } timestamp: 7023062361167421440 op_type: WRITE_OP write_request { tablet_id: "6b56adc64eba4898b80687a2679b70c8" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\327z\310\032D=\320\305\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""3378fa7c1709fa95" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023062361124511744 authz_token { token_data: "\010\317\346\313\261\006\"3\n\005slave\022*\n ae2e57387f5a496cb85e41e055010f5b\020\001\030\001 \001(\001" signature: "<redacted>""\036E\271\246\251l?\244z\200M\302\023m\274\234C\266S\355\334\213\305\360\363:\325\203Tx7\245$\006\264\370\254\351\330!\267\306\363\261\241v\000n`\252\303\337UN\344\235\005{eT\227\036%\220" signing_key_seq_num: 0 } } request_id { client_id: "d7e71b8df6ae46e0b8ae64a8ac9a4850" seq_no: 707 first_incomplete_seq_no: 707 attempt_no: 0 }
I20240502 01:53:54.590296  1775 raft_consensus.cc:2258] T 6b56adc64eba4898b80687a2679b70c8 P b191086bf5f2432b8d240510817ae5fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:54.595707  1775 ts_tablet_manager.cc:1875] T 6b56adc64eba4898b80687a2679b70c8 P b191086bf5f2432b8d240510817ae5fe: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:54.615391  1775 ts_tablet_manager.cc:1888] T 6b56adc64eba4898b80687a2679b70c8 P b191086bf5f2432b8d240510817ae5fe: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.228
I20240502 01:53:54.621099  1775 log.cc:1196] T 6b56adc64eba4898b80687a2679b70c8 P b191086bf5f2432b8d240510817ae5fe: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal/wals/6b56adc64eba4898b80687a2679b70c8
I20240502 01:53:54.624980 31490 catalog_manager.cc:4830] TS b191086bf5f2432b8d240510817ae5fe (127.25.109.129:42243): tablet 6b56adc64eba4898b80687a2679b70c8 (table rebalance_test_table_3 [id=ae2e57387f5a496cb85e41e055010f5b]) successfully deleted
W20240502 01:53:54.930327 31850 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.129:42243, user_credentials={real_user=slave}} blocked reactor thread for 182434us
I20240502 01:53:55.298213   497 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:55.379156 31896 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:53:55.497494 31764 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:53:55.558910   766 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:55.589542   630 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:55.663471 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:55.714040   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:56.043514 31784 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "82e5853a509c43df917f257186844ddc"
dest_uuid: "35f356f20e274db7bf020f78f1bc9ae7"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:36948
I20240502 01:53:56.043846 31784 raft_consensus.cc:595] T 82e5853a509c43df917f257186844ddc P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 LEADER]: Received request to transfer leadership
W20240502 01:53:56.070818 31741 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47916: Service unavailable: leader transfer in progress
W20240502 01:53:56.077487 31741 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47916: Service unavailable: leader transfer in progress
W20240502 01:53:56.101485 31741 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47916: Service unavailable: leader transfer in progress
W20240502 01:53:56.114887 31741 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47916: Service unavailable: leader transfer in progress
W20240502 01:53:56.127933 31742 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47916: Service unavailable: leader transfer in progress
W20240502 01:53:56.147609 31741 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47916: Service unavailable: leader transfer in progress
W20240502 01:53:56.167532 31741 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47916: Service unavailable: leader transfer in progress
W20240502 01:53:56.180488 31744 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47916: Service unavailable: leader transfer in progress
W20240502 01:53:56.193097 31744 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47916: Service unavailable: leader transfer in progress
W20240502 01:53:56.204185 31744 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47916: Service unavailable: leader transfer in progress
W20240502 01:53:56.225420 31744 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47916: Service unavailable: leader transfer in progress
W20240502 01:53:56.248822 31744 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47916: Service unavailable: leader transfer in progress
W20240502 01:53:56.265702 31741 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47916: Service unavailable: leader transfer in progress
W20240502 01:53:56.284870 31741 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47916: Service unavailable: leader transfer in progress
W20240502 01:53:56.305838 31741 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47916: Service unavailable: leader transfer in progress
W20240502 01:53:56.326442 31744 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47916: Service unavailable: leader transfer in progress
W20240502 01:53:56.353896 31744 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47916: Service unavailable: leader transfer in progress
W20240502 01:53:56.377010 31744 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47916: Service unavailable: leader transfer in progress
W20240502 01:53:56.404024 31744 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47916: Service unavailable: leader transfer in progress
W20240502 01:53:56.426242 31744 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47916: Service unavailable: leader transfer in progress
W20240502 01:53:56.467051 31744 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47916: Service unavailable: leader transfer in progress
W20240502 01:53:56.493916 31741 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47916: Service unavailable: leader transfer in progress
W20240502 01:53:56.522475 31741 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47916: Service unavailable: leader transfer in progress
I20240502 01:53:56.550173  1604 raft_consensus.cc:979] T 82e5853a509c43df917f257186844ddc P 35f356f20e274db7bf020f78f1bc9ae7: : Instructing follower 1864f9ea44c54742b090c594a5be3669 to start an election
I20240502 01:53:56.550385  1604 raft_consensus.cc:1067] T 82e5853a509c43df917f257186844ddc P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 LEADER]: Signalling peer 1864f9ea44c54742b090c594a5be3669 to start an election
W20240502 01:53:56.552537 31741 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47916: Service unavailable: leader transfer in progress
I20240502 01:53:56.553448 31915 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "82e5853a509c43df917f257186844ddc"
dest_uuid: "1864f9ea44c54742b090c594a5be3669"
 from {username='slave'} at 127.25.109.130:55707
I20240502 01:53:56.553723 31915 raft_consensus.cc:482] T 82e5853a509c43df917f257186844ddc P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:53:56.553858 31915 raft_consensus.cc:3037] T 82e5853a509c43df917f257186844ddc P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:56.557541 31915 raft_consensus.cc:504] T 82e5853a509c43df917f257186844ddc P 1864f9ea44c54742b090c594a5be3669 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 142 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } }
I20240502 01:53:56.560408 31915 leader_election.cc:288] T 82e5853a509c43df917f257186844ddc P 1864f9ea44c54742b090c594a5be3669 [CANDIDATE]: Term 2 election: Requested vote from peers 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130:41639), 0aea20ce2a1144519ed144383c016d78 (127.25.109.132:34649)
W20240502 01:53:56.583036 31741 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47916: Service unavailable: leader transfer in progress
W20240502 01:53:56.614037 31741 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47916: Service unavailable: leader transfer in progress
I20240502 01:53:56.627506 31784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82e5853a509c43df917f257186844ddc" candidate_uuid: "1864f9ea44c54742b090c594a5be3669" candidate_term: 2 candidate_status { last_received { term: 1 index: 212 } } ignore_live_leader: true dest_uuid: "35f356f20e274db7bf020f78f1bc9ae7"
I20240502 01:53:56.627801 31784 raft_consensus.cc:3032] T 82e5853a509c43df917f257186844ddc P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:53:56.627918 31784 raft_consensus.cc:729] T 82e5853a509c43df917f257186844ddc P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 35f356f20e274db7bf020f78f1bc9ae7, State: Running, Role: LEADER
I20240502 01:53:56.628319 31784 consensus_queue.cc:260] T 82e5853a509c43df917f257186844ddc P 35f356f20e274db7bf020f78f1bc9ae7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 212, Committed index: 212, Last appended: 1.212, Last appended by leader: 212, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 142 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } }
I20240502 01:53:56.628787 31784 raft_consensus.cc:3037] T 82e5853a509c43df917f257186844ddc P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:56.632050   516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82e5853a509c43df917f257186844ddc" candidate_uuid: "1864f9ea44c54742b090c594a5be3669" candidate_term: 2 candidate_status { last_received { term: 1 index: 212 } } ignore_live_leader: true dest_uuid: "0aea20ce2a1144519ed144383c016d78"
I20240502 01:53:56.632422   516 raft_consensus.cc:3037] T 82e5853a509c43df917f257186844ddc P 0aea20ce2a1144519ed144383c016d78 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:56.632652 31784 raft_consensus.cc:2454] T 82e5853a509c43df917f257186844ddc P 35f356f20e274db7bf020f78f1bc9ae7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1864f9ea44c54742b090c594a5be3669 in term 2.
I20240502 01:53:56.633167 31850 leader_election.cc:302] T 82e5853a509c43df917f257186844ddc P 1864f9ea44c54742b090c594a5be3669 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1864f9ea44c54742b090c594a5be3669, 35f356f20e274db7bf020f78f1bc9ae7; no voters: 
I20240502 01:53:56.634768  1786 raft_consensus.cc:2781] T 82e5853a509c43df917f257186844ddc P 1864f9ea44c54742b090c594a5be3669 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:53:56.634917  1786 raft_consensus.cc:686] T 82e5853a509c43df917f257186844ddc P 1864f9ea44c54742b090c594a5be3669 [term 2 LEADER]: Becoming Leader. State: Replica: 1864f9ea44c54742b090c594a5be3669, State: Running, Role: LEADER
I20240502 01:53:56.635334   516 raft_consensus.cc:2454] T 82e5853a509c43df917f257186844ddc P 0aea20ce2a1144519ed144383c016d78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1864f9ea44c54742b090c594a5be3669 in term 2.
I20240502 01:53:56.635267  1786 consensus_queue.cc:237] T 82e5853a509c43df917f257186844ddc P 1864f9ea44c54742b090c594a5be3669 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 212, Committed index: 212, Last appended: 1.212, Last appended by leader: 212, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 142 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } }
I20240502 01:53:56.639596 31502 catalog_manager.cc:5484] T 82e5853a509c43df917f257186844ddc P 1864f9ea44c54742b090c594a5be3669 reported cstate change: term changed from 1 to 2, leader changed from 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130) to 1864f9ea44c54742b090c594a5be3669 (127.25.109.131). New cstate: current_term: 2 leader_uuid: "1864f9ea44c54742b090c594a5be3669" committed_config { opid_index: 142 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } }
W20240502 01:53:56.645723 31741 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47916: Illegal state: replica 35f356f20e274db7bf020f78f1bc9ae7 is not leader of this config: current role FOLLOWER
I20240502 01:53:56.782045 31785 raft_consensus.cc:1261] T 82e5853a509c43df917f257186844ddc P 35f356f20e274db7bf020f78f1bc9ae7 [term 2 FOLLOWER]: Refusing update from remote peer 1864f9ea44c54742b090c594a5be3669: Log matching property violated. Preceding OpId in replica: term: 1 index: 212. Preceding OpId from leader: term: 2 index: 214. (index mismatch)
I20240502 01:53:56.783041   516 raft_consensus.cc:1261] T 82e5853a509c43df917f257186844ddc P 0aea20ce2a1144519ed144383c016d78 [term 2 FOLLOWER]: Refusing update from remote peer 1864f9ea44c54742b090c594a5be3669: Log matching property violated. Preceding OpId in replica: term: 1 index: 212. Preceding OpId from leader: term: 2 index: 214. (index mismatch)
I20240502 01:53:56.784405  1786 consensus_queue.cc:1035] T 82e5853a509c43df917f257186844ddc P 1864f9ea44c54742b090c594a5be3669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 213, Last known committed idx: 212, Time since last communication: 0.000s
I20240502 01:53:56.793915  1672 consensus_queue.cc:1035] T 82e5853a509c43df917f257186844ddc P 1864f9ea44c54742b090c594a5be3669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 213, Last known committed idx: 212, Time since last communication: 0.000s
I20240502 01:53:56.803694  1729 mvcc.cc:205] Tried to move back new op lower bound from 7023062371453263872 to 7023062370861281280. Current Snapshot: MvccSnapshot[applied={T|T < 7023062371453263872}]
I20240502 01:53:56.971239   497 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:57.049417 31896 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:53:57.098316 31764 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:53:57.120995   766 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:53:57.209868 31719 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.134:38633, user_credentials={real_user=slave}} blocked reactor thread for 57146.5us
I20240502 01:53:57.262688 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:57.317250   630 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:57.477663   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:53:58.019507 31849 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.132:34649, user_credentials={real_user=slave}} blocked reactor thread for 49023.1us
W20240502 01:53:58.595350 31716 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.132:34649, user_credentials={real_user=slave}} blocked reactor thread for 66818.4us
I20240502 01:53:58.886119 31914 consensus_queue.cc:237] T 82e5853a509c43df917f257186844ddc P 1864f9ea44c54742b090c594a5be3669 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 249, Committed index: 249, Last appended: 2.249, Last appended by leader: 212, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } }
I20240502 01:53:58.888573   515 raft_consensus.cc:1261] T 82e5853a509c43df917f257186844ddc P 0aea20ce2a1144519ed144383c016d78 [term 2 FOLLOWER]: Refusing update from remote peer 1864f9ea44c54742b090c594a5be3669: Log matching property violated. Preceding OpId in replica: term: 2 index: 249. Preceding OpId from leader: term: 2 index: 251. (index mismatch)
I20240502 01:53:58.889303  1672 consensus_queue.cc:1035] T 82e5853a509c43df917f257186844ddc P 1864f9ea44c54742b090c594a5be3669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 250, Last known committed idx: 249, Time since last communication: 0.000s
I20240502 01:53:58.909231  1672 raft_consensus.cc:2932] T 82e5853a509c43df917f257186844ddc P 1864f9ea44c54742b090c594a5be3669 [term 2 LEADER]: Committing config change with OpId 2.250: config changed from index 142 to 250, VOTER 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130) evicted. New config: { opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } }
I20240502 01:53:58.914100   516 raft_consensus.cc:2932] T 82e5853a509c43df917f257186844ddc P 0aea20ce2a1144519ed144383c016d78 [term 2 FOLLOWER]: Committing config change with OpId 2.250: config changed from index 142 to 250, VOTER 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130) evicted. New config: { opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } }
I20240502 01:53:58.940711 31500 catalog_manager.cc:5484] T 82e5853a509c43df917f257186844ddc P 1864f9ea44c54742b090c594a5be3669 reported cstate change: config changed from index 142 to 250, VOTER 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130) evicted. New cstate: current_term: 2 leader_uuid: "1864f9ea44c54742b090c594a5be3669" committed_config { opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } }
W20240502 01:53:59.063815 31716 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.132:34649, user_credentials={real_user=slave}} blocked reactor thread for 44353.8us
I20240502 01:53:59.132090 31764 tablet_service.cc:1514] Processing DeleteTablet for tablet 82e5853a509c43df917f257186844ddc with delete_type TABLET_DATA_TOMBSTONED (TS 35f356f20e274db7bf020f78f1bc9ae7 not found in new config with opid_index 250) from {username='slave'} at 127.0.0.1:46798
I20240502 01:53:59.342686  1973 tablet_replica.cc:330] T 82e5853a509c43df917f257186844ddc P 35f356f20e274db7bf020f78f1bc9ae7: stopping tablet replica
I20240502 01:53:59.343955  1973 raft_consensus.cc:2229] T 82e5853a509c43df917f257186844ddc P 35f356f20e274db7bf020f78f1bc9ae7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:59.346302  1973 raft_consensus.cc:2258] T 82e5853a509c43df917f257186844ddc P 35f356f20e274db7bf020f78f1bc9ae7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:59.354911  1973 ts_tablet_manager.cc:1875] T 82e5853a509c43df917f257186844ddc P 35f356f20e274db7bf020f78f1bc9ae7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:59.376435  1973 ts_tablet_manager.cc:1888] T 82e5853a509c43df917f257186844ddc P 35f356f20e274db7bf020f78f1bc9ae7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.249
I20240502 01:53:59.377936  1973 log.cc:1196] T 82e5853a509c43df917f257186844ddc P 35f356f20e274db7bf020f78f1bc9ae7: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal/wals/82e5853a509c43df917f257186844ddc
I20240502 01:53:59.379143 31490 catalog_manager.cc:4830] TS 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130:41639): tablet 82e5853a509c43df917f257186844ddc (table rebalance_test_table_4 [id=9a09fee1a01c4c1c836c0dab06832248]) successfully deleted
W20240502 01:53:59.626832 31582 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.130:41639, user_credentials={real_user=slave}} blocked reactor thread for 106762us
W20240502 01:53:59.855887 31581 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.132:34649, user_credentials={real_user=slave}} blocked reactor thread for 177952us
W20240502 01:54:00.162057 31716 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.131:42119, user_credentials={real_user=slave}} blocked reactor thread for 48024.7us
I20240502 01:54:00.759923 31764 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:00.817554   497 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:00.850919   766 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:00.906333 31896 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:54:00.927687   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:00.959136   630 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:01.137281 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:54:02.157472 31783 consensus_queue.cc:237] T fe2daac9229e44c9bee5f7fbaf886728 P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 325, Committed index: 325, Last appended: 1.325, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 326 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } }
I20240502 01:54:02.161005   783 raft_consensus.cc:1261] T fe2daac9229e44c9bee5f7fbaf886728 P 676092c80dd74040bb49a06db1ac209c [term 1 FOLLOWER]: Refusing update from remote peer 35f356f20e274db7bf020f78f1bc9ae7: Log matching property violated. Preceding OpId in replica: term: 1 index: 325. Preceding OpId from leader: term: 1 index: 326. (index mismatch)
I20240502 01:54:02.162843  1950 consensus_queue.cc:1035] T fe2daac9229e44c9bee5f7fbaf886728 P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 326, Last known committed idx: 325, Time since last communication: 0.000s
I20240502 01:54:02.169126  2092 raft_consensus.cc:2932] T fe2daac9229e44c9bee5f7fbaf886728 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 LEADER]: Committing config change with OpId 1.326: config changed from index 159 to 326, VOTER b191086bf5f2432b8d240510817ae5fe (127.25.109.129) evicted. New config: { opid_index: 326 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } } }
I20240502 01:54:02.170035   783 raft_consensus.cc:2932] T fe2daac9229e44c9bee5f7fbaf886728 P 676092c80dd74040bb49a06db1ac209c [term 1 FOLLOWER]: Committing config change with OpId 1.326: config changed from index 159 to 326, VOTER b191086bf5f2432b8d240510817ae5fe (127.25.109.129) evicted. New config: { opid_index: 326 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } } }
I20240502 01:54:02.174739 31500 catalog_manager.cc:5484] T fe2daac9229e44c9bee5f7fbaf886728 P 35f356f20e274db7bf020f78f1bc9ae7 reported cstate change: config changed from index 159 to 326, VOTER b191086bf5f2432b8d240510817ae5fe (127.25.109.129) evicted. New cstate: current_term: 1 leader_uuid: "35f356f20e274db7bf020f78f1bc9ae7" committed_config { opid_index: 326 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } } }
I20240502 01:54:02.300514 31628 tablet_service.cc:1514] Processing DeleteTablet for tablet fe2daac9229e44c9bee5f7fbaf886728 with delete_type TABLET_DATA_TOMBSTONED (TS b191086bf5f2432b8d240510817ae5fe not found in new config with opid_index 326) from {username='slave'} at 127.0.0.1:48324
I20240502 01:54:02.426302  2104 tablet_replica.cc:330] T fe2daac9229e44c9bee5f7fbaf886728 P b191086bf5f2432b8d240510817ae5fe: stopping tablet replica
I20240502 01:54:02.426614  2104 raft_consensus.cc:2229] T fe2daac9229e44c9bee5f7fbaf886728 P b191086bf5f2432b8d240510817ae5fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:02.426903  2104 raft_consensus.cc:2258] T fe2daac9229e44c9bee5f7fbaf886728 P b191086bf5f2432b8d240510817ae5fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:02.461309  2104 ts_tablet_manager.cc:1875] T fe2daac9229e44c9bee5f7fbaf886728 P b191086bf5f2432b8d240510817ae5fe: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 01:54:02.509814 31582 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.130:41639, user_credentials={real_user=slave}} blocked reactor thread for 153571us
I20240502 01:54:02.520947  2104 ts_tablet_manager.cc:1888] T fe2daac9229e44c9bee5f7fbaf886728 P b191086bf5f2432b8d240510817ae5fe: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.325
W20240502 01:54:02.522150   581 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.131:42119, user_credentials={real_user=slave}} blocked reactor thread for 46508.4us
I20240502 01:54:02.525648  2104 log.cc:1196] T fe2daac9229e44c9bee5f7fbaf886728 P b191086bf5f2432b8d240510817ae5fe: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal/wals/fe2daac9229e44c9bee5f7fbaf886728
I20240502 01:54:02.528014 31490 catalog_manager.cc:4830] TS b191086bf5f2432b8d240510817ae5fe (127.25.109.129:42243): tablet fe2daac9229e44c9bee5f7fbaf886728 (table rebalance_test_table_0 [id=c997c25d61cd4f25aad452793cf51df8]) successfully deleted
I20240502 01:54:03.446755 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:54:03.454867   766 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:03.480521   497 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:03.506404 31896 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:54:03.523823   630 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:03.534994 31764 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:03.856278   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:04.463892 31644 consensus_queue.cc:237] T 02b02a474b5e4f6ab173ced2dc21c03e P b191086bf5f2432b8d240510817ae5fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 383, Committed index: 383, Last appended: 1.383, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 384 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } }
I20240502 01:54:04.466878   919 raft_consensus.cc:1261] T 02b02a474b5e4f6ab173ced2dc21c03e P facf1046c8054d2d841cf875c63e29e2 [term 1 FOLLOWER]: Refusing update from remote peer b191086bf5f2432b8d240510817ae5fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 383. Preceding OpId from leader: term: 1 index: 384. (index mismatch)
I20240502 01:54:04.469333  2059 consensus_queue.cc:1035] T 02b02a474b5e4f6ab173ced2dc21c03e P b191086bf5f2432b8d240510817ae5fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 384, Last known committed idx: 383, Time since last communication: 0.001s
I20240502 01:54:04.473471  2059 raft_consensus.cc:2932] T 02b02a474b5e4f6ab173ced2dc21c03e P b191086bf5f2432b8d240510817ae5fe [term 1 LEADER]: Committing config change with OpId 1.384: config changed from index 159 to 384, VOTER 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130) evicted. New config: { opid_index: 384 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } } }
I20240502 01:54:04.479768   919 raft_consensus.cc:2932] T 02b02a474b5e4f6ab173ced2dc21c03e P facf1046c8054d2d841cf875c63e29e2 [term 1 FOLLOWER]: Committing config change with OpId 1.384: config changed from index 159 to 384, VOTER 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130) evicted. New config: { opid_index: 384 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } } }
I20240502 01:54:04.482949 31502 catalog_manager.cc:5484] T 02b02a474b5e4f6ab173ced2dc21c03e P b191086bf5f2432b8d240510817ae5fe reported cstate change: config changed from index 159 to 384, VOTER 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130) evicted. New cstate: current_term: 1 leader_uuid: "b191086bf5f2432b8d240510817ae5fe" committed_config { opid_index: 384 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } } }
I20240502 01:54:04.557766 31764 tablet_service.cc:1514] Processing DeleteTablet for tablet 02b02a474b5e4f6ab173ced2dc21c03e with delete_type TABLET_DATA_TOMBSTONED (TS 35f356f20e274db7bf020f78f1bc9ae7 not found in new config with opid_index 384) from {username='slave'} at 127.0.0.1:46798
I20240502 01:54:04.587191  2204 tablet_replica.cc:330] T 02b02a474b5e4f6ab173ced2dc21c03e P 35f356f20e274db7bf020f78f1bc9ae7: stopping tablet replica
I20240502 01:54:04.592464  2204 raft_consensus.cc:2229] T 02b02a474b5e4f6ab173ced2dc21c03e P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:04.592890  2204 raft_consensus.cc:2258] T 02b02a474b5e4f6ab173ced2dc21c03e P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:04.594453  2204 ts_tablet_manager.cc:1875] T 02b02a474b5e4f6ab173ced2dc21c03e P 35f356f20e274db7bf020f78f1bc9ae7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:04.602844  2204 ts_tablet_manager.cc:1888] T 02b02a474b5e4f6ab173ced2dc21c03e P 35f356f20e274db7bf020f78f1bc9ae7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.383
I20240502 01:54:04.603065  2204 log.cc:1196] T 02b02a474b5e4f6ab173ced2dc21c03e P 35f356f20e274db7bf020f78f1bc9ae7: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal/wals/02b02a474b5e4f6ab173ced2dc21c03e
I20240502 01:54:04.607597 31490 catalog_manager.cc:4830] TS 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130:41639): tablet 02b02a474b5e4f6ab173ced2dc21c03e (table rebalance_test_table_0 [id=c997c25d61cd4f25aad452793cf51df8]) successfully deleted
I20240502 01:54:05.618544 31896 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:54:05.634750   497 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:05.726542   766 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:05.916210 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:54:05.939826   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:05.998435   630 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:06.030346 31764 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:54:06.044816 31717 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.129:42243, user_credentials={real_user=slave}} blocked reactor thread for 71533us
I20240502 01:54:06.851599 31783 consensus_queue.cc:237] T 704700e8872b4675b41d6c172731a5e0 P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 394, Committed index: 394, Last appended: 1.394, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 395 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } }
I20240502 01:54:06.853566   783 raft_consensus.cc:1261] T 704700e8872b4675b41d6c172731a5e0 P 676092c80dd74040bb49a06db1ac209c [term 1 FOLLOWER]: Refusing update from remote peer 35f356f20e274db7bf020f78f1bc9ae7: Log matching property violated. Preceding OpId in replica: term: 1 index: 394. Preceding OpId from leader: term: 1 index: 395. (index mismatch)
I20240502 01:54:06.854674  1952 consensus_queue.cc:1035] T 704700e8872b4675b41d6c172731a5e0 P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 395, Last known committed idx: 394, Time since last communication: 0.000s
I20240502 01:54:06.858332  1949 raft_consensus.cc:2932] T 704700e8872b4675b41d6c172731a5e0 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 LEADER]: Committing config change with OpId 1.395: config changed from index 115 to 395, VOTER 1864f9ea44c54742b090c594a5be3669 (127.25.109.131) evicted. New config: { opid_index: 395 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } } }
I20240502 01:54:06.867408 31502 catalog_manager.cc:5484] T 704700e8872b4675b41d6c172731a5e0 P 35f356f20e274db7bf020f78f1bc9ae7 reported cstate change: config changed from index 115 to 395, VOTER 1864f9ea44c54742b090c594a5be3669 (127.25.109.131) evicted. New cstate: current_term: 1 leader_uuid: "35f356f20e274db7bf020f78f1bc9ae7" committed_config { opid_index: 395 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } } }
I20240502 01:54:06.877744   783 raft_consensus.cc:2932] T 704700e8872b4675b41d6c172731a5e0 P 676092c80dd74040bb49a06db1ac209c [term 1 FOLLOWER]: Committing config change with OpId 1.395: config changed from index 115 to 395, VOTER 1864f9ea44c54742b090c594a5be3669 (127.25.109.131) evicted. New config: { opid_index: 395 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } } }
I20240502 01:54:07.031394 31896 tablet_service.cc:1514] Processing DeleteTablet for tablet 704700e8872b4675b41d6c172731a5e0 with delete_type TABLET_DATA_TOMBSTONED (TS 1864f9ea44c54742b090c594a5be3669 not found in new config with opid_index 395) from {username='slave'} at 127.0.0.1:40670
I20240502 01:54:07.069968  2307 tablet_replica.cc:330] T 704700e8872b4675b41d6c172731a5e0 P 1864f9ea44c54742b090c594a5be3669: stopping tablet replica
I20240502 01:54:07.070668  2307 raft_consensus.cc:2229] T 704700e8872b4675b41d6c172731a5e0 P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:07.079439  2307 pending_rounds.cc:62] T 704700e8872b4675b41d6c172731a5e0 P 1864f9ea44c54742b090c594a5be3669: Trying to abort 1 pending ops.
I20240502 01:54:07.079701  2307 pending_rounds.cc:68] T 704700e8872b4675b41d6c172731a5e0 P 1864f9ea44c54742b090c594a5be3669: Aborting op as it isn't in flight: id { term: 1 index: 394 } timestamp: 7023062412507541504 op_type: WRITE_OP write_request { tablet_id: "704700e8872b4675b41d6c172731a5e0" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\2437~v\374fE\347\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""aaf447d89731d987" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023062412441341952 authz_token { token_data: "\010\317\346\313\261\006\"3\n\005slave\022*\n 7340ad22b7574a6d81f7eaa977a06414\020\001\030\001 \001(\001" signature: "<redacted>""DZ\253\261\tz_TS\032=\363\031+\230Oz\316\300\014<vF\3003\345yd\366\273\235\376\376\306\017w\205ZCQT\372\264\253h\t\242\364\016&\212_@\026\316!\213\024P\240\210D_w" signing_key_seq_num: 0 } } request_id { client_id: "d3814d865ee94a2caa44c3b6e0ff24fc" seq_no: 1161 first_incomplete_seq_no: 1161 attempt_no: 0 }
I20240502 01:54:07.080902  2307 raft_consensus.cc:2258] T 704700e8872b4675b41d6c172731a5e0 P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:07.088580  2307 ts_tablet_manager.cc:1875] T 704700e8872b4675b41d6c172731a5e0 P 1864f9ea44c54742b090c594a5be3669: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:07.096506  2307 ts_tablet_manager.cc:1888] T 704700e8872b4675b41d6c172731a5e0 P 1864f9ea44c54742b090c594a5be3669: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.394
I20240502 01:54:07.096716  2307 log.cc:1196] T 704700e8872b4675b41d6c172731a5e0 P 1864f9ea44c54742b090c594a5be3669: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/wals/704700e8872b4675b41d6c172731a5e0
I20240502 01:54:07.097975 31489 catalog_manager.cc:4830] TS 1864f9ea44c54742b090c594a5be3669 (127.25.109.131:42119): tablet 704700e8872b4675b41d6c172731a5e0 (table rebalance_test_table_2 [id=7340ad22b7574a6d81f7eaa977a06414]) successfully deleted
I20240502 01:54:07.983747   497 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:08.005964 31896 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:54:08.171485 31764 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:08.323737   630 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:08.340080   766 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:08.427820 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:54:08.448011   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:08.886703 31783 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4a00f35083c9404ea2ea877826f2fe77"
dest_uuid: "35f356f20e274db7bf020f78f1bc9ae7"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:48192
I20240502 01:54:08.886901 31783 raft_consensus.cc:595] T 4a00f35083c9404ea2ea877826f2fe77 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:54:08.892499  1949 raft_consensus.cc:979] T 4a00f35083c9404ea2ea877826f2fe77 P 35f356f20e274db7bf020f78f1bc9ae7: : Instructing follower 676092c80dd74040bb49a06db1ac209c to start an election
I20240502 01:54:08.892704  1949 raft_consensus.cc:1067] T 4a00f35083c9404ea2ea877826f2fe77 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 LEADER]: Signalling peer 676092c80dd74040bb49a06db1ac209c to start an election
W20240502 01:54:08.921802 31741 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47916: Service unavailable: leader transfer in progress
W20240502 01:54:08.929677 31741 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47916: Service unavailable: leader transfer in progress
I20240502 01:54:08.935138   786 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4a00f35083c9404ea2ea877826f2fe77"
dest_uuid: "676092c80dd74040bb49a06db1ac209c"
 from {username='slave'} at 127.25.109.130:36309
I20240502 01:54:08.935617   786 raft_consensus.cc:482] T 4a00f35083c9404ea2ea877826f2fe77 P 676092c80dd74040bb49a06db1ac209c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
W20240502 01:54:08.936852 31741 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47916: Service unavailable: leader transfer in progress
I20240502 01:54:08.937783   786 raft_consensus.cc:3037] T 4a00f35083c9404ea2ea877826f2fe77 P 676092c80dd74040bb49a06db1ac209c [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:54:08.953230 31744 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47916: Service unavailable: leader transfer in progress
I20240502 01:54:08.958151   786 raft_consensus.cc:504] T 4a00f35083c9404ea2ea877826f2fe77 P 676092c80dd74040bb49a06db1ac209c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } }
I20240502 01:54:08.961584   786 leader_election.cc:288] T 4a00f35083c9404ea2ea877826f2fe77 P 676092c80dd74040bb49a06db1ac209c [CANDIDATE]: Term 2 election: Requested vote from peers b191086bf5f2432b8d240510817ae5fe (127.25.109.129:42243), 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130:41639)
W20240502 01:54:08.962786 31741 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47916: Service unavailable: leader transfer in progress
W20240502 01:54:08.973444 31741 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47916: Service unavailable: leader transfer in progress
W20240502 01:54:08.982435 31741 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47916: Service unavailable: leader transfer in progress
I20240502 01:54:08.985965 31783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a00f35083c9404ea2ea877826f2fe77" candidate_uuid: "676092c80dd74040bb49a06db1ac209c" candidate_term: 2 candidate_status { last_received { term: 1 index: 404 } } ignore_live_leader: true dest_uuid: "35f356f20e274db7bf020f78f1bc9ae7"
I20240502 01:54:08.986214 31783 raft_consensus.cc:3032] T 4a00f35083c9404ea2ea877826f2fe77 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:08.986286 31783 raft_consensus.cc:729] T 4a00f35083c9404ea2ea877826f2fe77 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 35f356f20e274db7bf020f78f1bc9ae7, State: Running, Role: LEADER
I20240502 01:54:08.986532 31783 consensus_queue.cc:260] T 4a00f35083c9404ea2ea877826f2fe77 P 35f356f20e274db7bf020f78f1bc9ae7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 404, Committed index: 404, Last appended: 1.404, Last appended by leader: 404, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } }
I20240502 01:54:08.986886 31783 raft_consensus.cc:3037] T 4a00f35083c9404ea2ea877826f2fe77 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:08.989948 31783 raft_consensus.cc:2454] T 4a00f35083c9404ea2ea877826f2fe77 P 35f356f20e274db7bf020f78f1bc9ae7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 676092c80dd74040bb49a06db1ac209c in term 2.
I20240502 01:54:08.991744   719 leader_election.cc:302] T 4a00f35083c9404ea2ea877826f2fe77 P 676092c80dd74040bb49a06db1ac209c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35f356f20e274db7bf020f78f1bc9ae7, 676092c80dd74040bb49a06db1ac209c; no voters: 
W20240502 01:54:08.992172 31744 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47916: Illegal state: replica 35f356f20e274db7bf020f78f1bc9ae7 is not leader of this config: current role FOLLOWER
I20240502 01:54:09.004528  2385 raft_consensus.cc:2781] T 4a00f35083c9404ea2ea877826f2fe77 P 676092c80dd74040bb49a06db1ac209c [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:09.014926 31647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a00f35083c9404ea2ea877826f2fe77" candidate_uuid: "676092c80dd74040bb49a06db1ac209c" candidate_term: 2 candidate_status { last_received { term: 1 index: 404 } } ignore_live_leader: true dest_uuid: "b191086bf5f2432b8d240510817ae5fe"
I20240502 01:54:09.015173 31647 raft_consensus.cc:3037] T 4a00f35083c9404ea2ea877826f2fe77 P b191086bf5f2432b8d240510817ae5fe [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:09.017921 31647 raft_consensus.cc:2454] T 4a00f35083c9404ea2ea877826f2fe77 P b191086bf5f2432b8d240510817ae5fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 676092c80dd74040bb49a06db1ac209c in term 2.
I20240502 01:54:09.027649  2385 raft_consensus.cc:686] T 4a00f35083c9404ea2ea877826f2fe77 P 676092c80dd74040bb49a06db1ac209c [term 2 LEADER]: Becoming Leader. State: Replica: 676092c80dd74040bb49a06db1ac209c, State: Running, Role: LEADER
I20240502 01:54:09.028409  2385 consensus_queue.cc:237] T 4a00f35083c9404ea2ea877826f2fe77 P 676092c80dd74040bb49a06db1ac209c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 404, Committed index: 404, Last appended: 1.404, Last appended by leader: 404, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } }
I20240502 01:54:09.038739 31502 catalog_manager.cc:5484] T 4a00f35083c9404ea2ea877826f2fe77 P 676092c80dd74040bb49a06db1ac209c reported cstate change: term changed from 1 to 2, leader changed from 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130) to 676092c80dd74040bb49a06db1ac209c (127.25.109.134). New cstate: current_term: 2 leader_uuid: "676092c80dd74040bb49a06db1ac209c" committed_config { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } } }
W20240502 01:54:09.093834 31609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40470: Illegal state: replica b191086bf5f2432b8d240510817ae5fe is not leader of this config: current role FOLLOWER
I20240502 01:54:09.164134 31647 raft_consensus.cc:1261] T 4a00f35083c9404ea2ea877826f2fe77 P b191086bf5f2432b8d240510817ae5fe [term 2 FOLLOWER]: Refusing update from remote peer 676092c80dd74040bb49a06db1ac209c: Log matching property violated. Preceding OpId in replica: term: 1 index: 404. Preceding OpId from leader: term: 2 index: 406. (index mismatch)
I20240502 01:54:09.165582 31783 raft_consensus.cc:1261] T 4a00f35083c9404ea2ea877826f2fe77 P 35f356f20e274db7bf020f78f1bc9ae7 [term 2 FOLLOWER]: Refusing update from remote peer 676092c80dd74040bb49a06db1ac209c: Log matching property violated. Preceding OpId in replica: term: 1 index: 404. Preceding OpId from leader: term: 2 index: 406. (index mismatch)
I20240502 01:54:09.168980  2385 consensus_queue.cc:1035] T 4a00f35083c9404ea2ea877826f2fe77 P 676092c80dd74040bb49a06db1ac209c [LEADER]: Connected to new peer: Peer: permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 405, Last known committed idx: 404, Time since last communication: 0.000s
I20240502 01:54:09.169564  2387 consensus_queue.cc:1035] T 4a00f35083c9404ea2ea877826f2fe77 P 676092c80dd74040bb49a06db1ac209c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 405, Last known committed idx: 404, Time since last communication: 0.000s
I20240502 01:54:09.180452  2388 mvcc.cc:205] Tried to move back new op lower bound from 7023062422165118976 to 7023062421640114176. Current Snapshot: MvccSnapshot[applied={T|T < 7023062422165118976 or (T in {7023062422165118976})}]
W20240502 01:54:10.003674 31582 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.130:41639, user_credentials={real_user=slave}} blocked reactor thread for 69870.7us
I20240502 01:54:10.111588 31896 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:54:10.166358   497 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:10.204506 31764 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:54:10.236472   766 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:10.246925   630 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:54:10.383265 31716 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.132:34649, user_credentials={real_user=slave}} blocked reactor thread for 56848.5us
I20240502 01:54:10.388576   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:10.430238 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:54:11.449643   784 consensus_queue.cc:237] T 4a00f35083c9404ea2ea877826f2fe77 P 676092c80dd74040bb49a06db1ac209c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 442, Committed index: 442, Last appended: 2.443, Last appended by leader: 404, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 444 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } }
I20240502 01:54:11.463162 31647 raft_consensus.cc:1261] T 4a00f35083c9404ea2ea877826f2fe77 P b191086bf5f2432b8d240510817ae5fe [term 2 FOLLOWER]: Refusing update from remote peer 676092c80dd74040bb49a06db1ac209c: Log matching property violated. Preceding OpId in replica: term: 2 index: 443. Preceding OpId from leader: term: 2 index: 444. (index mismatch)
I20240502 01:54:11.470775  2416 consensus_queue.cc:1035] T 4a00f35083c9404ea2ea877826f2fe77 P 676092c80dd74040bb49a06db1ac209c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 444, Last known committed idx: 442, Time since last communication: 0.000s
I20240502 01:54:11.513020  2416 raft_consensus.cc:2932] T 4a00f35083c9404ea2ea877826f2fe77 P 676092c80dd74040bb49a06db1ac209c [term 2 LEADER]: Committing config change with OpId 2.444: config changed from index 68 to 444, VOTER 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130) evicted. New config: { opid_index: 444 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } } }
I20240502 01:54:11.577013 31502 catalog_manager.cc:5484] T 4a00f35083c9404ea2ea877826f2fe77 P 676092c80dd74040bb49a06db1ac209c reported cstate change: config changed from index 68 to 444, VOTER 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130) evicted. New cstate: current_term: 2 leader_uuid: "676092c80dd74040bb49a06db1ac209c" committed_config { opid_index: 444 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } } }
I20240502 01:54:11.585417 31647 raft_consensus.cc:2932] T 4a00f35083c9404ea2ea877826f2fe77 P b191086bf5f2432b8d240510817ae5fe [term 2 FOLLOWER]: Committing config change with OpId 2.444: config changed from index 68 to 444, VOTER 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130) evicted. New config: { opid_index: 444 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } } }
I20240502 01:54:11.757169 31764 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a00f35083c9404ea2ea877826f2fe77 with delete_type TABLET_DATA_TOMBSTONED (TS 35f356f20e274db7bf020f78f1bc9ae7 not found in new config with opid_index 444) from {username='slave'} at 127.0.0.1:46798
I20240502 01:54:11.780426  2504 tablet_replica.cc:330] T 4a00f35083c9404ea2ea877826f2fe77 P 35f356f20e274db7bf020f78f1bc9ae7: stopping tablet replica
I20240502 01:54:11.781028  2504 raft_consensus.cc:2229] T 4a00f35083c9404ea2ea877826f2fe77 P 35f356f20e274db7bf020f78f1bc9ae7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:11.791855  2504 pending_rounds.cc:62] T 4a00f35083c9404ea2ea877826f2fe77 P 35f356f20e274db7bf020f78f1bc9ae7: Trying to abort 1 pending ops.
I20240502 01:54:11.792104  2504 pending_rounds.cc:68] T 4a00f35083c9404ea2ea877826f2fe77 P 35f356f20e274db7bf020f78f1bc9ae7: Aborting op as it isn't in flight: id { term: 2 index: 440 } timestamp: 7023062431024513024 op_type: WRITE_OP write_request { tablet_id: "4a00f35083c9404ea2ea877826f2fe77" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000A\377l\024\273\325v_\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""d9161a32f423ec9e" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023062430950969344 authz_token { token_data: "\010\317\346\313\261\006\"3\n\005slave\022*\n 9a09fee1a01c4c1c836c0dab06832248\020\001\030\001 \001(\001" signature: "<redacted>""gVZ\357\257\242\021=\224\242\222\316\032\355P\324V\002h\313\334\352\350\016Q\3648jj\024\n4\334\221\212\006\232\200\025#<m\321\236\\\231\001\277\225\016e\203\240\244\240\232-_\224\236VO\034\203" signing_key_seq_num: 0 } } request_id { client_id: "c88dc9bd359f436a968c2fffb2a457a7" seq_no: 1319 first_incomplete_seq_no: 1319 attempt_no: 0 }
I20240502 01:54:11.792894  2504 raft_consensus.cc:2258] T 4a00f35083c9404ea2ea877826f2fe77 P 35f356f20e274db7bf020f78f1bc9ae7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:11.794247  2504 ts_tablet_manager.cc:1875] T 4a00f35083c9404ea2ea877826f2fe77 P 35f356f20e274db7bf020f78f1bc9ae7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:11.804582  2504 ts_tablet_manager.cc:1888] T 4a00f35083c9404ea2ea877826f2fe77 P 35f356f20e274db7bf020f78f1bc9ae7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.440
I20240502 01:54:11.805104  2504 log.cc:1196] T 4a00f35083c9404ea2ea877826f2fe77 P 35f356f20e274db7bf020f78f1bc9ae7: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal/wals/4a00f35083c9404ea2ea877826f2fe77
I20240502 01:54:11.820474 31490 catalog_manager.cc:4830] TS 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130:41639): tablet 4a00f35083c9404ea2ea877826f2fe77 (table rebalance_test_table_4 [id=9a09fee1a01c4c1c836c0dab06832248]) successfully deleted
I20240502 01:54:12.967283   766 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:12.974203   497 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:12.999073   630 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:13.023190   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:13.128435 31896 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:54:13.144461 31764 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:54:13.156391 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:54:13.353340 31914 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5575decad4be448e835db06b570c69a8"
dest_uuid: "1864f9ea44c54742b090c594a5be3669"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:40328
I20240502 01:54:13.353708 31914 raft_consensus.cc:595] T 5575decad4be448e835db06b570c69a8 P 1864f9ea44c54742b090c594a5be3669 [term 1 LEADER]: Received request to transfer leadership
W20240502 01:54:13.388845 31874 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38414: Service unavailable: leader transfer in progress
W20240502 01:54:13.396677 31874 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38414: Service unavailable: leader transfer in progress
W20240502 01:54:13.407375 31876 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38414: Service unavailable: leader transfer in progress
W20240502 01:54:13.417644 31876 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38414: Service unavailable: leader transfer in progress
W20240502 01:54:13.427626 31876 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38414: Service unavailable: leader transfer in progress
W20240502 01:54:13.453655 31876 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38414: Service unavailable: leader transfer in progress
W20240502 01:54:13.464844 31876 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38414: Service unavailable: leader transfer in progress
W20240502 01:54:13.476127 31876 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38414: Service unavailable: leader transfer in progress
W20240502 01:54:13.489244 31874 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38414: Service unavailable: leader transfer in progress
W20240502 01:54:13.516479 31876 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38414: Service unavailable: leader transfer in progress
W20240502 01:54:13.531338 31876 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38414: Service unavailable: leader transfer in progress
W20240502 01:54:13.548264 31876 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38414: Service unavailable: leader transfer in progress
W20240502 01:54:13.566087 31874 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38414: Service unavailable: leader transfer in progress
W20240502 01:54:13.586097 31876 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38414: Service unavailable: leader transfer in progress
W20240502 01:54:13.605291 31876 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38414: Service unavailable: leader transfer in progress
W20240502 01:54:13.624619 31874 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38414: Service unavailable: leader transfer in progress
W20240502 01:54:13.648363 31874 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38414: Service unavailable: leader transfer in progress
W20240502 01:54:13.675498 31874 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38414: Service unavailable: leader transfer in progress
W20240502 01:54:13.701388 31874 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38414: Service unavailable: leader transfer in progress
W20240502 01:54:13.723548 31874 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38414: Service unavailable: leader transfer in progress
W20240502 01:54:13.749267 31874 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38414: Service unavailable: leader transfer in progress
W20240502 01:54:13.781651 31874 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38414: Service unavailable: leader transfer in progress
I20240502 01:54:13.808404  2072 raft_consensus.cc:979] T 5575decad4be448e835db06b570c69a8 P 1864f9ea44c54742b090c594a5be3669: : Instructing follower a506969c1a5346c09c19a01f6cbc1a8b to start an election
I20240502 01:54:13.808748  2072 raft_consensus.cc:1067] T 5575decad4be448e835db06b570c69a8 P 1864f9ea44c54742b090c594a5be3669 [term 1 LEADER]: Signalling peer a506969c1a5346c09c19a01f6cbc1a8b to start an election
I20240502 01:54:13.810288   651 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5575decad4be448e835db06b570c69a8"
dest_uuid: "a506969c1a5346c09c19a01f6cbc1a8b"
 from {username='slave'} at 127.25.109.131:48649
I20240502 01:54:13.810602   651 raft_consensus.cc:482] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:13.810743   651 raft_consensus.cc:3037] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:54:13.811702 31874 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38414: Service unavailable: leader transfer in progress
I20240502 01:54:13.813570   651 raft_consensus.cc:504] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } }
I20240502 01:54:13.816607   651 leader_election.cc:288] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b [CANDIDATE]: Term 2 election: Requested vote from peers b191086bf5f2432b8d240510817ae5fe (127.25.109.129:42243), 1864f9ea44c54742b090c594a5be3669 (127.25.109.131:42119)
I20240502 01:54:13.816965 31915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5575decad4be448e835db06b570c69a8" candidate_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" candidate_term: 2 candidate_status { last_received { term: 1 index: 518 } } ignore_live_leader: true dest_uuid: "1864f9ea44c54742b090c594a5be3669"
I20240502 01:54:13.817191 31915 raft_consensus.cc:3032] T 5575decad4be448e835db06b570c69a8 P 1864f9ea44c54742b090c594a5be3669 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:13.817266 31915 raft_consensus.cc:729] T 5575decad4be448e835db06b570c69a8 P 1864f9ea44c54742b090c594a5be3669 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1864f9ea44c54742b090c594a5be3669, State: Running, Role: LEADER
I20240502 01:54:13.817564 31915 consensus_queue.cc:260] T 5575decad4be448e835db06b570c69a8 P 1864f9ea44c54742b090c594a5be3669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 518, Committed index: 518, Last appended: 1.518, Last appended by leader: 518, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } }
I20240502 01:54:13.817834 31915 raft_consensus.cc:3037] T 5575decad4be448e835db06b570c69a8 P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:13.830418 31915 raft_consensus.cc:2454] T 5575decad4be448e835db06b570c69a8 P 1864f9ea44c54742b090c594a5be3669 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a506969c1a5346c09c19a01f6cbc1a8b in term 2.
I20240502 01:54:13.844995   581 leader_election.cc:302] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1864f9ea44c54742b090c594a5be3669, a506969c1a5346c09c19a01f6cbc1a8b; no voters: 
I20240502 01:54:13.845400  2595 raft_consensus.cc:2781] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:13.845569  2595 raft_consensus.cc:686] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b [term 2 LEADER]: Becoming Leader. State: Replica: a506969c1a5346c09c19a01f6cbc1a8b, State: Running, Role: LEADER
I20240502 01:54:13.845885  2595 consensus_queue.cc:237] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 518, Committed index: 518, Last appended: 1.518, Last appended by leader: 518, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } }
I20240502 01:54:13.849298   497 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:13.854171 31502 catalog_manager.cc:5484] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b reported cstate change: term changed from 1 to 2, leader changed from 1864f9ea44c54742b090c594a5be3669 (127.25.109.131) to a506969c1a5346c09c19a01f6cbc1a8b (127.25.109.133). New cstate: current_term: 2 leader_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" committed_config { opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } }
W20240502 01:54:13.859449 31874 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38414: Illegal state: replica 1864f9ea44c54742b090c594a5be3669 is not leader of this config: current role FOLLOWER
I20240502 01:54:13.890841 31648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5575decad4be448e835db06b570c69a8" candidate_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" candidate_term: 2 candidate_status { last_received { term: 1 index: 518 } } ignore_live_leader: true dest_uuid: "b191086bf5f2432b8d240510817ae5fe"
I20240502 01:54:13.891069 31648 raft_consensus.cc:3037] T 5575decad4be448e835db06b570c69a8 P b191086bf5f2432b8d240510817ae5fe [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:13.899958 31648 raft_consensus.cc:2454] T 5575decad4be448e835db06b570c69a8 P b191086bf5f2432b8d240510817ae5fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a506969c1a5346c09c19a01f6cbc1a8b in term 2.
W20240502 01:54:14.009819 31609 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40504: Illegal state: replica b191086bf5f2432b8d240510817ae5fe is not leader of this config: current role FOLLOWER
I20240502 01:54:14.029208 31764 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:54:14.057371   630 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:54:14.065263   766 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:14.104682 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:54:14.113451   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:14.132221 31896 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:54:14.145776 31915 raft_consensus.cc:1261] T 5575decad4be448e835db06b570c69a8 P 1864f9ea44c54742b090c594a5be3669 [term 2 FOLLOWER]: Refusing update from remote peer a506969c1a5346c09c19a01f6cbc1a8b: Log matching property violated. Preceding OpId in replica: term: 1 index: 518. Preceding OpId from leader: term: 2 index: 520. (index mismatch)
I20240502 01:54:14.146602  2597 consensus_queue.cc:1035] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 519, Last known committed idx: 518, Time since last communication: 0.000s
I20240502 01:54:14.149741 31644 raft_consensus.cc:1261] T 5575decad4be448e835db06b570c69a8 P b191086bf5f2432b8d240510817ae5fe [term 2 FOLLOWER]: Refusing update from remote peer a506969c1a5346c09c19a01f6cbc1a8b: Log matching property violated. Preceding OpId in replica: term: 1 index: 518. Preceding OpId from leader: term: 2 index: 520. (index mismatch)
I20240502 01:54:14.152240  2595 consensus_queue.cc:1035] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 519, Last known committed idx: 518, Time since last communication: 0.000s
I20240502 01:54:14.154735  2566 mvcc.cc:205] Tried to move back new op lower bound from 7023062442566799360 to 7023062441355804672. Current Snapshot: MvccSnapshot[applied={T|T < 7023062442566799360}]
I20240502 01:54:14.178857  2629 mvcc.cc:205] Tried to move back new op lower bound from 7023062442566799360 to 7023062441355804672. Current Snapshot: MvccSnapshot[applied={T|T < 7023062442566799360}]
I20240502 01:54:15.888149 31896 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:54:15.898658   497 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:15.977334   630 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:54:15.991798   766 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:15.996624 31764 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:54:16.040663   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:16.113605 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 01:54:16.934844 31582 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.130:41639, user_credentials={real_user=slave}} blocked reactor thread for 60784.5us
I20240502 01:54:16.998035 31912 consensus_queue.cc:237] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 432, Committed index: 432, Last appended: 1.432, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 433 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } }
I20240502 01:54:17.002200   919 raft_consensus.cc:1261] T 559610821917470fbcf5641d36acf409 P facf1046c8054d2d841cf875c63e29e2 [term 1 FOLLOWER]: Refusing update from remote peer 1864f9ea44c54742b090c594a5be3669: Log matching property violated. Preceding OpId in replica: term: 1 index: 432. Preceding OpId from leader: term: 1 index: 433. (index mismatch)
I20240502 01:54:17.003062  2510 consensus_queue.cc:1035] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 433, Last known committed idx: 431, Time since last communication: 0.000s
I20240502 01:54:17.006367  2510 raft_consensus.cc:2932] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669 [term 1 LEADER]: Committing config change with OpId 1.433: config changed from index 71 to 433, VOTER b191086bf5f2432b8d240510817ae5fe (127.25.109.129) evicted. New config: { opid_index: 433 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } } }
I20240502 01:54:17.007417   920 raft_consensus.cc:2932] T 559610821917470fbcf5641d36acf409 P facf1046c8054d2d841cf875c63e29e2 [term 1 FOLLOWER]: Committing config change with OpId 1.433: config changed from index 71 to 433, VOTER b191086bf5f2432b8d240510817ae5fe (127.25.109.129) evicted. New config: { opid_index: 433 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } } }
I20240502 01:54:17.012734 31500 catalog_manager.cc:5484] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669 reported cstate change: config changed from index 71 to 433, VOTER b191086bf5f2432b8d240510817ae5fe (127.25.109.129) evicted. New cstate: current_term: 1 leader_uuid: "1864f9ea44c54742b090c594a5be3669" committed_config { opid_index: 433 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } } }
I20240502 01:54:17.090824 31628 tablet_service.cc:1514] Processing DeleteTablet for tablet 559610821917470fbcf5641d36acf409 with delete_type TABLET_DATA_TOMBSTONED (TS b191086bf5f2432b8d240510817ae5fe not found in new config with opid_index 433) from {username='slave'} at 127.0.0.1:48324
I20240502 01:54:17.097007  2750 tablet_replica.cc:330] T 559610821917470fbcf5641d36acf409 P b191086bf5f2432b8d240510817ae5fe: stopping tablet replica
I20240502 01:54:17.097287  2750 raft_consensus.cc:2229] T 559610821917470fbcf5641d36acf409 P b191086bf5f2432b8d240510817ae5fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:17.097569  2750 raft_consensus.cc:2258] T 559610821917470fbcf5641d36acf409 P b191086bf5f2432b8d240510817ae5fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:17.098995  2750 ts_tablet_manager.cc:1875] T 559610821917470fbcf5641d36acf409 P b191086bf5f2432b8d240510817ae5fe: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:17.105262  2750 ts_tablet_manager.cc:1888] T 559610821917470fbcf5641d36acf409 P b191086bf5f2432b8d240510817ae5fe: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.432
I20240502 01:54:17.105383  2750 log.cc:1196] T 559610821917470fbcf5641d36acf409 P b191086bf5f2432b8d240510817ae5fe: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal/wals/559610821917470fbcf5641d36acf409
I20240502 01:54:17.106093 31490 catalog_manager.cc:4830] TS b191086bf5f2432b8d240510817ae5fe (127.25.109.129:42243): tablet 559610821917470fbcf5641d36acf409 (table rebalance_test_table_1 [id=76d2ee30f2bb480087daf717aa077ba2]) successfully deleted
I20240502 01:54:18.633777   497 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:18.641839   630 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:54:18.646385 31764 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:54:18.713111   766 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:18.743445   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:18.871517 31896 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:54:18.891184 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 01:54:19.334246   581 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.131:42119, user_credentials={real_user=slave}} blocked reactor thread for 129717us
I20240502 01:54:19.919548 31784 consensus_queue.cc:237] T e714cba54201433caa218073097c1e6a P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 596, Committed index: 596, Last appended: 1.596, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 597 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } }
I20240502 01:54:19.923508   916 raft_consensus.cc:1261] T e714cba54201433caa218073097c1e6a P facf1046c8054d2d841cf875c63e29e2 [term 1 FOLLOWER]: Refusing update from remote peer 35f356f20e274db7bf020f78f1bc9ae7: Log matching property violated. Preceding OpId in replica: term: 1 index: 596. Preceding OpId from leader: term: 1 index: 597. (index mismatch)
I20240502 01:54:19.924162  1949 consensus_queue.cc:1035] T e714cba54201433caa218073097c1e6a P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 597, Last known committed idx: 596, Time since last communication: 0.000s
I20240502 01:54:19.927115  1750 raft_consensus.cc:2932] T e714cba54201433caa218073097c1e6a P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 LEADER]: Committing config change with OpId 1.597: config changed from index 135 to 597, VOTER 1864f9ea44c54742b090c594a5be3669 (127.25.109.131) evicted. New config: { opid_index: 597 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } } }
I20240502 01:54:19.929157   919 raft_consensus.cc:2932] T e714cba54201433caa218073097c1e6a P facf1046c8054d2d841cf875c63e29e2 [term 1 FOLLOWER]: Committing config change with OpId 1.597: config changed from index 135 to 597, VOTER 1864f9ea44c54742b090c594a5be3669 (127.25.109.131) evicted. New config: { opid_index: 597 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } } }
I20240502 01:54:19.940798 31501 catalog_manager.cc:5484] T e714cba54201433caa218073097c1e6a P 35f356f20e274db7bf020f78f1bc9ae7 reported cstate change: config changed from index 135 to 597, VOTER 1864f9ea44c54742b090c594a5be3669 (127.25.109.131) evicted. New cstate: current_term: 1 leader_uuid: "35f356f20e274db7bf020f78f1bc9ae7" committed_config { opid_index: 597 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } } }
I20240502 01:54:20.062243 31896 tablet_service.cc:1514] Processing DeleteTablet for tablet e714cba54201433caa218073097c1e6a with delete_type TABLET_DATA_TOMBSTONED (TS 1864f9ea44c54742b090c594a5be3669 not found in new config with opid_index 597) from {username='slave'} at 127.0.0.1:40670
I20240502 01:54:20.084328  2861 tablet_replica.cc:330] T e714cba54201433caa218073097c1e6a P 1864f9ea44c54742b090c594a5be3669: stopping tablet replica
I20240502 01:54:20.084633  2861 raft_consensus.cc:2229] T e714cba54201433caa218073097c1e6a P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:20.084964  2861 raft_consensus.cc:2258] T e714cba54201433caa218073097c1e6a P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:20.086979  2861 ts_tablet_manager.cc:1875] T e714cba54201433caa218073097c1e6a P 1864f9ea44c54742b090c594a5be3669: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:20.095132  2861 ts_tablet_manager.cc:1888] T e714cba54201433caa218073097c1e6a P 1864f9ea44c54742b090c594a5be3669: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.596
I20240502 01:54:20.095369  2861 log.cc:1196] T e714cba54201433caa218073097c1e6a P 1864f9ea44c54742b090c594a5be3669: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/wals/e714cba54201433caa218073097c1e6a
I20240502 01:54:20.102828 31489 catalog_manager.cc:4830] TS 1864f9ea44c54742b090c594a5be3669 (127.25.109.131:42119): tablet e714cba54201433caa218073097c1e6a (table rebalance_test_table_4 [id=9a09fee1a01c4c1c836c0dab06832248]) successfully deleted
I20240502 01:54:20.743544   497 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:20.857299 31896 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:54:20.896958 31764 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:54:20.924360   766 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:21.110502 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:54:21.168738   630 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:54:21.212193   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:54:21.219655 31584 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.135:37253, user_credentials={real_user=slave}} blocked reactor thread for 61351.7us
I20240502 01:54:21.802662 31913 consensus_queue.cc:237] T 1313391ae5b2440d93445deda2642a14 P 1864f9ea44c54742b090c594a5be3669 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 582, Committed index: 582, Last appended: 1.582, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 583 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } }
I20240502 01:54:21.805682   919 raft_consensus.cc:1261] T 1313391ae5b2440d93445deda2642a14 P facf1046c8054d2d841cf875c63e29e2 [term 1 FOLLOWER]: Refusing update from remote peer 1864f9ea44c54742b090c594a5be3669: Log matching property violated. Preceding OpId in replica: term: 1 index: 582. Preceding OpId from leader: term: 1 index: 583. (index mismatch)
I20240502 01:54:21.808773  2674 consensus_queue.cc:1035] T 1313391ae5b2440d93445deda2642a14 P 1864f9ea44c54742b090c594a5be3669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 583, Last known committed idx: 582, Time since last communication: 0.000s
I20240502 01:54:21.814432  2291 raft_consensus.cc:2932] T 1313391ae5b2440d93445deda2642a14 P 1864f9ea44c54742b090c594a5be3669 [term 1 LEADER]: Committing config change with OpId 1.583: config changed from index 50 to 583, VOTER b191086bf5f2432b8d240510817ae5fe (127.25.109.129) evicted. New config: { opid_index: 583 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } } }
I20240502 01:54:21.844666 31500 catalog_manager.cc:5484] T 1313391ae5b2440d93445deda2642a14 P 1864f9ea44c54742b090c594a5be3669 reported cstate change: config changed from index 50 to 583, VOTER b191086bf5f2432b8d240510817ae5fe (127.25.109.129) evicted. New cstate: current_term: 1 leader_uuid: "1864f9ea44c54742b090c594a5be3669" committed_config { opid_index: 583 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } } }
W20240502 01:54:21.877547 31852 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.135:37253, user_credentials={real_user=slave}} blocked reactor thread for 63616.4us
I20240502 01:54:21.878636   919 raft_consensus.cc:2932] T 1313391ae5b2440d93445deda2642a14 P facf1046c8054d2d841cf875c63e29e2 [term 1 FOLLOWER]: Committing config change with OpId 1.583: config changed from index 50 to 583, VOTER b191086bf5f2432b8d240510817ae5fe (127.25.109.129) evicted. New config: { opid_index: 583 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } } }
I20240502 01:54:21.968775 31628 tablet_service.cc:1514] Processing DeleteTablet for tablet 1313391ae5b2440d93445deda2642a14 with delete_type TABLET_DATA_TOMBSTONED (TS b191086bf5f2432b8d240510817ae5fe not found in new config with opid_index 583) from {username='slave'} at 127.0.0.1:48324
I20240502 01:54:22.001057  2951 tablet_replica.cc:330] T 1313391ae5b2440d93445deda2642a14 P b191086bf5f2432b8d240510817ae5fe: stopping tablet replica
I20240502 01:54:22.003449  2951 raft_consensus.cc:2229] T 1313391ae5b2440d93445deda2642a14 P b191086bf5f2432b8d240510817ae5fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:22.004766  2951 raft_consensus.cc:2258] T 1313391ae5b2440d93445deda2642a14 P b191086bf5f2432b8d240510817ae5fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:22.008414  2951 ts_tablet_manager.cc:1875] T 1313391ae5b2440d93445deda2642a14 P b191086bf5f2432b8d240510817ae5fe: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:22.029531  2951 ts_tablet_manager.cc:1888] T 1313391ae5b2440d93445deda2642a14 P b191086bf5f2432b8d240510817ae5fe: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.582
I20240502 01:54:22.030864  2951 log.cc:1196] T 1313391ae5b2440d93445deda2642a14 P b191086bf5f2432b8d240510817ae5fe: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal/wals/1313391ae5b2440d93445deda2642a14
I20240502 01:54:22.033378 31490 catalog_manager.cc:4830] TS b191086bf5f2432b8d240510817ae5fe (127.25.109.129:42243): tablet 1313391ae5b2440d93445deda2642a14 (table rebalance_test_table_3 [id=ae2e57387f5a496cb85e41e055010f5b]) successfully deleted
I20240502 01:54:22.915396   497 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:22.993304 31764 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:54:23.039505 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:54:23.094439   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:23.095432   766 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:23.098475 31896 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:54:23.126695   630 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 01:54:23.357992 31852 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.135:37253, user_credentials={real_user=slave}} blocked reactor thread for 62734.4us
I20240502 01:54:23.732120 31784 consensus_queue.cc:237] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 588, Committed index: 588, Last appended: 1.588, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 589 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } }
I20240502 01:54:23.763963   517 raft_consensus.cc:1261] T 238e56f8d83748e8b2cf1100e92db4c6 P 0aea20ce2a1144519ed144383c016d78 [term 1 FOLLOWER]: Refusing update from remote peer 35f356f20e274db7bf020f78f1bc9ae7: Log matching property violated. Preceding OpId in replica: term: 1 index: 588. Preceding OpId from leader: term: 1 index: 589. (index mismatch)
I20240502 01:54:23.768622  2970 consensus_queue.cc:1035] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 589, Last known committed idx: 588, Time since last communication: 0.000s
I20240502 01:54:23.774053  2970 raft_consensus.cc:2932] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 LEADER]: Committing config change with OpId 1.589: config changed from index 64 to 589, VOTER 1864f9ea44c54742b090c594a5be3669 (127.25.109.131) evicted. New config: { opid_index: 589 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } }
I20240502 01:54:23.780745   514 raft_consensus.cc:2932] T 238e56f8d83748e8b2cf1100e92db4c6 P 0aea20ce2a1144519ed144383c016d78 [term 1 FOLLOWER]: Committing config change with OpId 1.589: config changed from index 64 to 589, VOTER 1864f9ea44c54742b090c594a5be3669 (127.25.109.131) evicted. New config: { opid_index: 589 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } }
I20240502 01:54:23.786597 31502 catalog_manager.cc:5484] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7 reported cstate change: config changed from index 64 to 589, VOTER 1864f9ea44c54742b090c594a5be3669 (127.25.109.131) evicted. New cstate: current_term: 1 leader_uuid: "35f356f20e274db7bf020f78f1bc9ae7" committed_config { opid_index: 589 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } }
I20240502 01:54:23.870172 31896 tablet_service.cc:1514] Processing DeleteTablet for tablet 238e56f8d83748e8b2cf1100e92db4c6 with delete_type TABLET_DATA_TOMBSTONED (TS 1864f9ea44c54742b090c594a5be3669 not found in new config with opid_index 589) from {username='slave'} at 127.0.0.1:40670
I20240502 01:54:23.874008  3050 tablet_replica.cc:330] T 238e56f8d83748e8b2cf1100e92db4c6 P 1864f9ea44c54742b090c594a5be3669: stopping tablet replica
I20240502 01:54:23.874349  3050 raft_consensus.cc:2229] T 238e56f8d83748e8b2cf1100e92db4c6 P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:23.874662  3050 raft_consensus.cc:2258] T 238e56f8d83748e8b2cf1100e92db4c6 P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:23.877900  3050 ts_tablet_manager.cc:1875] T 238e56f8d83748e8b2cf1100e92db4c6 P 1864f9ea44c54742b090c594a5be3669: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:23.907168  3050 ts_tablet_manager.cc:1888] T 238e56f8d83748e8b2cf1100e92db4c6 P 1864f9ea44c54742b090c594a5be3669: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.588
I20240502 01:54:23.907368  3050 log.cc:1196] T 238e56f8d83748e8b2cf1100e92db4c6 P 1864f9ea44c54742b090c594a5be3669: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/wals/238e56f8d83748e8b2cf1100e92db4c6
I20240502 01:54:23.910396 31489 catalog_manager.cc:4830] TS 1864f9ea44c54742b090c594a5be3669 (127.25.109.131:42119): tablet 238e56f8d83748e8b2cf1100e92db4c6 (table rebalance_test_table_1 [id=76d2ee30f2bb480087daf717aa077ba2]) successfully deleted
I20240502 01:54:24.584136 31764 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:54:24.616257 31896 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:54:24.644827   766 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:24.645431   497 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:24.752280   630 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:54:24.793781   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:24.837774 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:54:25.548756 31783 consensus_queue.cc:237] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 665, Committed index: 665, Last appended: 2.665, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 666 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } }
I20240502 01:54:25.554104   514 raft_consensus.cc:1261] T 10aa745ed48d46a3b0558d35dee6a807 P 0aea20ce2a1144519ed144383c016d78 [term 2 FOLLOWER]: Refusing update from remote peer 35f356f20e274db7bf020f78f1bc9ae7: Log matching property violated. Preceding OpId in replica: term: 2 index: 665. Preceding OpId from leader: term: 2 index: 666. (index mismatch)
I20240502 01:54:25.562742  2970 consensus_queue.cc:1035] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 666, Last known committed idx: 665, Time since last communication: 0.000s
I20240502 01:54:25.567487  2970 raft_consensus.cc:2932] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [term 2 LEADER]: Committing config change with OpId 2.666: config changed from index 149 to 666, VOTER b191086bf5f2432b8d240510817ae5fe (127.25.109.129) evicted. New config: { opid_index: 666 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } }
I20240502 01:54:25.574934 31502 catalog_manager.cc:5484] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 reported cstate change: config changed from index 149 to 666, VOTER b191086bf5f2432b8d240510817ae5fe (127.25.109.129) evicted. New cstate: current_term: 2 leader_uuid: "35f356f20e274db7bf020f78f1bc9ae7" committed_config { opid_index: 666 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } }
I20240502 01:54:25.575191   516 raft_consensus.cc:2932] T 10aa745ed48d46a3b0558d35dee6a807 P 0aea20ce2a1144519ed144383c016d78 [term 2 FOLLOWER]: Committing config change with OpId 2.666: config changed from index 149 to 666, VOTER b191086bf5f2432b8d240510817ae5fe (127.25.109.129) evicted. New config: { opid_index: 666 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } }
I20240502 01:54:25.696913 31628 tablet_service.cc:1514] Processing DeleteTablet for tablet 10aa745ed48d46a3b0558d35dee6a807 with delete_type TABLET_DATA_TOMBSTONED (TS b191086bf5f2432b8d240510817ae5fe not found in new config with opid_index 666) from {username='slave'} at 127.0.0.1:48324
I20240502 01:54:25.735636  3132 tablet_replica.cc:330] T 10aa745ed48d46a3b0558d35dee6a807 P b191086bf5f2432b8d240510817ae5fe: stopping tablet replica
I20240502 01:54:25.743098  3132 raft_consensus.cc:2229] T 10aa745ed48d46a3b0558d35dee6a807 P b191086bf5f2432b8d240510817ae5fe [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:25.743448  3132 raft_consensus.cc:2258] T 10aa745ed48d46a3b0558d35dee6a807 P b191086bf5f2432b8d240510817ae5fe [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:25.752377  3132 ts_tablet_manager.cc:1875] T 10aa745ed48d46a3b0558d35dee6a807 P b191086bf5f2432b8d240510817ae5fe: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:25.780994  3132 ts_tablet_manager.cc:1888] T 10aa745ed48d46a3b0558d35dee6a807 P b191086bf5f2432b8d240510817ae5fe: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.665
I20240502 01:54:25.781266  3132 log.cc:1196] T 10aa745ed48d46a3b0558d35dee6a807 P b191086bf5f2432b8d240510817ae5fe: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal/wals/10aa745ed48d46a3b0558d35dee6a807
I20240502 01:54:25.791384 31490 catalog_manager.cc:4830] TS b191086bf5f2432b8d240510817ae5fe (127.25.109.129:42243): tablet 10aa745ed48d46a3b0558d35dee6a807 (table rebalance_test_table_0 [id=c997c25d61cd4f25aad452793cf51df8]) successfully deleted
I20240502 01:54:26.601671 31896 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:54:26.610834 31764 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:54:26.645824   497 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:26.741019   766 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:26.829891   630 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:54:26.936241   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:27.017179 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:54:27.485558 31643 consensus_queue.cc:237] T 86f2e43af7fb4e02bb853f0748d62ddb P b191086bf5f2432b8d240510817ae5fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 677, Committed index: 677, Last appended: 1.677, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 678 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } }
I20240502 01:54:27.489689   516 raft_consensus.cc:1261] T 86f2e43af7fb4e02bb853f0748d62ddb P 0aea20ce2a1144519ed144383c016d78 [term 1 FOLLOWER]: Refusing update from remote peer b191086bf5f2432b8d240510817ae5fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 677. Preceding OpId from leader: term: 1 index: 678. (index mismatch)
I20240502 01:54:27.491375  3212 consensus_queue.cc:1035] T 86f2e43af7fb4e02bb853f0748d62ddb P b191086bf5f2432b8d240510817ae5fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 678, Last known committed idx: 677, Time since last communication: 0.000s
I20240502 01:54:27.495831  3212 raft_consensus.cc:2932] T 86f2e43af7fb4e02bb853f0748d62ddb P b191086bf5f2432b8d240510817ae5fe [term 1 LEADER]: Committing config change with OpId 1.678: config changed from index 113 to 678, VOTER 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130) evicted. New config: { opid_index: 678 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } }
I20240502 01:54:27.501094   513 raft_consensus.cc:2932] T 86f2e43af7fb4e02bb853f0748d62ddb P 0aea20ce2a1144519ed144383c016d78 [term 1 FOLLOWER]: Committing config change with OpId 1.678: config changed from index 113 to 678, VOTER 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130) evicted. New config: { opid_index: 678 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } }
I20240502 01:54:27.513319 31499 catalog_manager.cc:5484] T 86f2e43af7fb4e02bb853f0748d62ddb P b191086bf5f2432b8d240510817ae5fe reported cstate change: config changed from index 113 to 678, VOTER 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130) evicted. New cstate: current_term: 1 leader_uuid: "b191086bf5f2432b8d240510817ae5fe" committed_config { opid_index: 678 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } }
I20240502 01:54:27.630663 31764 tablet_service.cc:1514] Processing DeleteTablet for tablet 86f2e43af7fb4e02bb853f0748d62ddb with delete_type TABLET_DATA_TOMBSTONED (TS 35f356f20e274db7bf020f78f1bc9ae7 not found in new config with opid_index 678) from {username='slave'} at 127.0.0.1:46798
I20240502 01:54:27.648824  3227 tablet_replica.cc:330] T 86f2e43af7fb4e02bb853f0748d62ddb P 35f356f20e274db7bf020f78f1bc9ae7: stopping tablet replica
I20240502 01:54:27.649189  3227 raft_consensus.cc:2229] T 86f2e43af7fb4e02bb853f0748d62ddb P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:27.649457  3227 raft_consensus.cc:2258] T 86f2e43af7fb4e02bb853f0748d62ddb P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:27.650611  3227 ts_tablet_manager.cc:1875] T 86f2e43af7fb4e02bb853f0748d62ddb P 35f356f20e274db7bf020f78f1bc9ae7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:27.659798  3227 ts_tablet_manager.cc:1888] T 86f2e43af7fb4e02bb853f0748d62ddb P 35f356f20e274db7bf020f78f1bc9ae7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.677
I20240502 01:54:27.660046  3227 log.cc:1196] T 86f2e43af7fb4e02bb853f0748d62ddb P 35f356f20e274db7bf020f78f1bc9ae7: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal/wals/86f2e43af7fb4e02bb853f0748d62ddb
I20240502 01:54:27.662117 31490 catalog_manager.cc:4830] TS 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130:41639): tablet 86f2e43af7fb4e02bb853f0748d62ddb (table rebalance_test_table_2 [id=7340ad22b7574a6d81f7eaa977a06414]) successfully deleted
W20240502 01:54:27.960427 31584 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.135:37253, user_credentials={real_user=slave}} blocked reactor thread for 70412.3us
I20240502 01:54:29.557425   497 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:29.671243   766 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:29.678217 31896 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:54:29.691737   630 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:54:29.696349 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:54:29.763759 31764 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:54:29.854118   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:30.341503   651 consensus_queue.cc:237] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 813, Committed index: 813, Last appended: 2.813, Last appended by leader: 518, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 814 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } }
I20240502 01:54:30.346046 31643 raft_consensus.cc:1261] T 5575decad4be448e835db06b570c69a8 P b191086bf5f2432b8d240510817ae5fe [term 2 FOLLOWER]: Refusing update from remote peer a506969c1a5346c09c19a01f6cbc1a8b: Log matching property violated. Preceding OpId in replica: term: 2 index: 813. Preceding OpId from leader: term: 2 index: 814. (index mismatch)
I20240502 01:54:30.348084  3236 consensus_queue.cc:1035] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 814, Last known committed idx: 813, Time since last communication: 0.000s
I20240502 01:54:30.353540  3236 raft_consensus.cc:2932] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b [term 2 LEADER]: Committing config change with OpId 2.814: config changed from index 100 to 814, VOTER 1864f9ea44c54742b090c594a5be3669 (127.25.109.131) evicted. New config: { opid_index: 814 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } }
I20240502 01:54:30.368273 31646 raft_consensus.cc:2932] T 5575decad4be448e835db06b570c69a8 P b191086bf5f2432b8d240510817ae5fe [term 2 FOLLOWER]: Committing config change with OpId 2.814: config changed from index 100 to 814, VOTER 1864f9ea44c54742b090c594a5be3669 (127.25.109.131) evicted. New config: { opid_index: 814 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } }
I20240502 01:54:30.374296 31501 catalog_manager.cc:5484] T 5575decad4be448e835db06b570c69a8 P b191086bf5f2432b8d240510817ae5fe reported cstate change: config changed from index 100 to 814, VOTER 1864f9ea44c54742b090c594a5be3669 (127.25.109.131) evicted. New cstate: current_term: 2 leader_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" committed_config { opid_index: 814 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } }
I20240502 01:54:30.453136 31896 tablet_service.cc:1514] Processing DeleteTablet for tablet 5575decad4be448e835db06b570c69a8 with delete_type TABLET_DATA_TOMBSTONED (TS 1864f9ea44c54742b090c594a5be3669 not found in new config with opid_index 814) from {username='slave'} at 127.0.0.1:40670
I20240502 01:54:30.505221  3368 tablet_replica.cc:330] T 5575decad4be448e835db06b570c69a8 P 1864f9ea44c54742b090c594a5be3669: stopping tablet replica
I20240502 01:54:30.505460  3368 raft_consensus.cc:2229] T 5575decad4be448e835db06b570c69a8 P 1864f9ea44c54742b090c594a5be3669 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:30.505683  3368 raft_consensus.cc:2258] T 5575decad4be448e835db06b570c69a8 P 1864f9ea44c54742b090c594a5be3669 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:30.507053  3368 ts_tablet_manager.cc:1875] T 5575decad4be448e835db06b570c69a8 P 1864f9ea44c54742b090c594a5be3669: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:30.520695  3368 ts_tablet_manager.cc:1888] T 5575decad4be448e835db06b570c69a8 P 1864f9ea44c54742b090c594a5be3669: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.813
I20240502 01:54:30.524030  3368 log.cc:1196] T 5575decad4be448e835db06b570c69a8 P 1864f9ea44c54742b090c594a5be3669: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/wals/5575decad4be448e835db06b570c69a8
I20240502 01:54:30.526125 31489 catalog_manager.cc:4830] TS 1864f9ea44c54742b090c594a5be3669 (127.25.109.131:42119): tablet 5575decad4be448e835db06b570c69a8 (table rebalance_test_table_3 [id=ae2e57387f5a496cb85e41e055010f5b]) successfully deleted
I20240502 01:54:31.882560 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:54:31.896536   630 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:54:31.897491   497 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:31.926824 31896 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:54:31.937812 31764 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:54:31.953950   766 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:31.958132   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:32.456060 31912 consensus_queue.cc:237] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 707, Committed index: 707, Last appended: 1.707, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 708 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } }
W20240502 01:54:32.469867 31852 consensus_peers.cc:485] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669 -> Peer a506969c1a5346c09c19a01f6cbc1a8b (127.25.109.133:46379): Couldn't send request to peer a506969c1a5346c09c19a01f6cbc1a8b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 559610821917470fbcf5641d36acf409. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:32.472869   919 raft_consensus.cc:1261] T 559610821917470fbcf5641d36acf409 P facf1046c8054d2d841cf875c63e29e2 [term 1 FOLLOWER]: Refusing update from remote peer 1864f9ea44c54742b090c594a5be3669: Log matching property violated. Preceding OpId in replica: term: 1 index: 707. Preceding OpId from leader: term: 1 index: 708. (index mismatch)
I20240502 01:54:32.497337  3014 consensus_queue.cc:1035] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 708, Last known committed idx: 707, Time since last communication: 0.000s
I20240502 01:54:32.506459  3014 raft_consensus.cc:2932] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669 [term 1 LEADER]: Committing config change with OpId 1.708: config changed from index 433 to 708, VOTER a506969c1a5346c09c19a01f6cbc1a8b (127.25.109.133) added. New config: { opid_index: 708 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } }
I20240502 01:54:32.507647   917 raft_consensus.cc:2932] T 559610821917470fbcf5641d36acf409 P facf1046c8054d2d841cf875c63e29e2 [term 1 FOLLOWER]: Committing config change with OpId 1.708: config changed from index 433 to 708, VOTER a506969c1a5346c09c19a01f6cbc1a8b (127.25.109.133) added. New config: { opid_index: 708 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } }
I20240502 01:54:32.512912 31501 catalog_manager.cc:5484] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669 reported cstate change: config changed from index 433 to 708, VOTER a506969c1a5346c09c19a01f6cbc1a8b (127.25.109.133) added. New cstate: current_term: 1 leader_uuid: "1864f9ea44c54742b090c594a5be3669" committed_config { opid_index: 708 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } }
I20240502 01:54:32.726388 31783 consensus_queue.cc:237] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 761, Committed index: 761, Last appended: 2.761, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 762 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } }
I20240502 01:54:32.730716   514 raft_consensus.cc:1261] T 10aa745ed48d46a3b0558d35dee6a807 P 0aea20ce2a1144519ed144383c016d78 [term 2 FOLLOWER]: Refusing update from remote peer 35f356f20e274db7bf020f78f1bc9ae7: Log matching property violated. Preceding OpId in replica: term: 2 index: 761. Preceding OpId from leader: term: 2 index: 762. (index mismatch)
I20240502 01:54:32.731400  3473 consensus_queue.cc:1035] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 762, Last known committed idx: 761, Time since last communication: 0.000s
I20240502 01:54:32.733698  3473 raft_consensus.cc:2932] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [term 2 LEADER]: Committing config change with OpId 2.762: config changed from index 666 to 762, VOTER a506969c1a5346c09c19a01f6cbc1a8b (127.25.109.133) added. New config: { opid_index: 762 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } }
I20240502 01:54:32.738785   514 raft_consensus.cc:2932] T 10aa745ed48d46a3b0558d35dee6a807 P 0aea20ce2a1144519ed144383c016d78 [term 2 FOLLOWER]: Committing config change with OpId 2.762: config changed from index 666 to 762, VOTER a506969c1a5346c09c19a01f6cbc1a8b (127.25.109.133) added. New config: { opid_index: 762 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } }
I20240502 01:54:32.740007 31499 catalog_manager.cc:5484] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 reported cstate change: config changed from index 666 to 762, VOTER a506969c1a5346c09c19a01f6cbc1a8b (127.25.109.133) added. New cstate: current_term: 2 leader_uuid: "35f356f20e274db7bf020f78f1bc9ae7" committed_config { opid_index: 762 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } }
W20240502 01:54:32.831350 31719 consensus_peers.cc:485] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 -> Peer a506969c1a5346c09c19a01f6cbc1a8b (127.25.109.133:46379): Couldn't send request to peer a506969c1a5346c09c19a01f6cbc1a8b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 10aa745ed48d46a3b0558d35dee6a807. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:32.879536 31783 consensus_queue.cc:237] T 65576af2c4bb4f25b4d14963cbe65ca7 P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 792, Committed index: 792, Last appended: 1.792, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 793 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } }
I20240502 01:54:32.881953 31912 raft_consensus.cc:1261] T 65576af2c4bb4f25b4d14963cbe65ca7 P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Refusing update from remote peer 35f356f20e274db7bf020f78f1bc9ae7: Log matching property violated. Preceding OpId in replica: term: 1 index: 792. Preceding OpId from leader: term: 1 index: 794. (index mismatch)
I20240502 01:54:32.882594  3296 consensus_queue.cc:1035] T 65576af2c4bb4f25b4d14963cbe65ca7 P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 793, Last known committed idx: 792, Time since last communication: 0.000s
W20240502 01:54:32.883144 31719 consensus_peers.cc:485] T 65576af2c4bb4f25b4d14963cbe65ca7 P 35f356f20e274db7bf020f78f1bc9ae7 -> Peer a506969c1a5346c09c19a01f6cbc1a8b (127.25.109.133:46379): Couldn't send request to peer a506969c1a5346c09c19a01f6cbc1a8b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 65576af2c4bb4f25b4d14963cbe65ca7. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:32.888531  3473 raft_consensus.cc:2932] T 65576af2c4bb4f25b4d14963cbe65ca7 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 LEADER]: Committing config change with OpId 1.793: config changed from index -1 to 793, VOTER a506969c1a5346c09c19a01f6cbc1a8b (127.25.109.133) added. New config: { opid_index: 793 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } }
I20240502 01:54:32.895411 31499 catalog_manager.cc:5484] T 65576af2c4bb4f25b4d14963cbe65ca7 P 35f356f20e274db7bf020f78f1bc9ae7 reported cstate change: config changed from index -1 to 793, VOTER a506969c1a5346c09c19a01f6cbc1a8b (127.25.109.133) added. New cstate: current_term: 1 leader_uuid: "35f356f20e274db7bf020f78f1bc9ae7" committed_config { opid_index: 793 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } }
I20240502 01:54:32.903643 31912 raft_consensus.cc:2932] T 65576af2c4bb4f25b4d14963cbe65ca7 P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Committing config change with OpId 1.793: config changed from index -1 to 793, VOTER a506969c1a5346c09c19a01f6cbc1a8b (127.25.109.133) added. New config: { opid_index: 793 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } }
I20240502 01:54:33.050694  3543 ts_tablet_manager.cc:902] T 559610821917470fbcf5641d36acf409 P a506969c1a5346c09c19a01f6cbc1a8b: Initiating tablet copy from peer 1864f9ea44c54742b090c594a5be3669 (127.25.109.131:42119)
I20240502 01:54:33.071408  3543 tablet_copy_client.cc:308] T 559610821917470fbcf5641d36acf409 P a506969c1a5346c09c19a01f6cbc1a8b: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.131:42119
I20240502 01:54:33.079703 31936 tablet_copy_service.cc:140] P 1864f9ea44c54742b090c594a5be3669: Received BeginTabletCopySession request for tablet 559610821917470fbcf5641d36acf409 from peer a506969c1a5346c09c19a01f6cbc1a8b ({username='slave'} at 127.25.109.133:44189)
I20240502 01:54:33.079918 31936 tablet_copy_service.cc:161] P 1864f9ea44c54742b090c594a5be3669: Beginning new tablet copy session on tablet 559610821917470fbcf5641d36acf409 from peer a506969c1a5346c09c19a01f6cbc1a8b at {username='slave'} at 127.25.109.133:44189: session id = a506969c1a5346c09c19a01f6cbc1a8b-559610821917470fbcf5641d36acf409
I20240502 01:54:33.081864 31936 tablet_copy_source_session.cc:205] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:33.083245  3543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 559610821917470fbcf5641d36acf409. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:33.090171  3543 tablet_copy_client.cc:783] T 559610821917470fbcf5641d36acf409 P a506969c1a5346c09c19a01f6cbc1a8b: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:33.090498  3543 tablet_copy_client.cc:647] T 559610821917470fbcf5641d36acf409 P a506969c1a5346c09c19a01f6cbc1a8b: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:33.098019  3543 tablet_copy_client.cc:523] T 559610821917470fbcf5641d36acf409 P a506969c1a5346c09c19a01f6cbc1a8b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:33.103204  3543 tablet_bootstrap.cc:492] T 559610821917470fbcf5641d36acf409 P a506969c1a5346c09c19a01f6cbc1a8b: Bootstrap starting.
I20240502 01:54:33.245043  3550 ts_tablet_manager.cc:902] T 10aa745ed48d46a3b0558d35dee6a807 P a506969c1a5346c09c19a01f6cbc1a8b: Initiating tablet copy from peer 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130:41639)
I20240502 01:54:33.414161  3550 tablet_copy_client.cc:308] T 10aa745ed48d46a3b0558d35dee6a807 P a506969c1a5346c09c19a01f6cbc1a8b: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.130:41639
I20240502 01:54:33.415148  3562 ts_tablet_manager.cc:902] T 65576af2c4bb4f25b4d14963cbe65ca7 P a506969c1a5346c09c19a01f6cbc1a8b: Initiating tablet copy from peer 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130:41639)
I20240502 01:54:33.469357 31805 tablet_copy_service.cc:140] P 35f356f20e274db7bf020f78f1bc9ae7: Received BeginTabletCopySession request for tablet 10aa745ed48d46a3b0558d35dee6a807 from peer a506969c1a5346c09c19a01f6cbc1a8b ({username='slave'} at 127.25.109.133:60367)
I20240502 01:54:33.469558 31805 tablet_copy_service.cc:161] P 35f356f20e274db7bf020f78f1bc9ae7: Beginning new tablet copy session on tablet 10aa745ed48d46a3b0558d35dee6a807 from peer a506969c1a5346c09c19a01f6cbc1a8b at {username='slave'} at 127.25.109.133:60367: session id = a506969c1a5346c09c19a01f6cbc1a8b-10aa745ed48d46a3b0558d35dee6a807
I20240502 01:54:33.472558 31805 tablet_copy_source_session.cc:205] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:33.475178  3550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10aa745ed48d46a3b0558d35dee6a807. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:33.514703  3550 tablet_copy_client.cc:783] T 10aa745ed48d46a3b0558d35dee6a807 P a506969c1a5346c09c19a01f6cbc1a8b: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:33.515098  3550 tablet_copy_client.cc:647] T 10aa745ed48d46a3b0558d35dee6a807 P a506969c1a5346c09c19a01f6cbc1a8b: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:33.529271  3550 tablet_copy_client.cc:523] T 10aa745ed48d46a3b0558d35dee6a807 P a506969c1a5346c09c19a01f6cbc1a8b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:33.570461  3562 tablet_copy_client.cc:308] T 65576af2c4bb4f25b4d14963cbe65ca7 P a506969c1a5346c09c19a01f6cbc1a8b: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.130:41639
I20240502 01:54:33.572603 31805 tablet_copy_service.cc:140] P 35f356f20e274db7bf020f78f1bc9ae7: Received BeginTabletCopySession request for tablet 65576af2c4bb4f25b4d14963cbe65ca7 from peer a506969c1a5346c09c19a01f6cbc1a8b ({username='slave'} at 127.25.109.133:60367)
I20240502 01:54:33.572772 31805 tablet_copy_service.cc:161] P 35f356f20e274db7bf020f78f1bc9ae7: Beginning new tablet copy session on tablet 65576af2c4bb4f25b4d14963cbe65ca7 from peer a506969c1a5346c09c19a01f6cbc1a8b at {username='slave'} at 127.25.109.133:60367: session id = a506969c1a5346c09c19a01f6cbc1a8b-65576af2c4bb4f25b4d14963cbe65ca7
I20240502 01:54:33.575765 31805 tablet_copy_source_session.cc:205] T 65576af2c4bb4f25b4d14963cbe65ca7 P 35f356f20e274db7bf020f78f1bc9ae7: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:33.579326  3550 tablet_bootstrap.cc:492] T 10aa745ed48d46a3b0558d35dee6a807 P a506969c1a5346c09c19a01f6cbc1a8b: Bootstrap starting.
I20240502 01:54:33.602685  3562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65576af2c4bb4f25b4d14963cbe65ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:33.634073  3562 tablet_copy_client.cc:783] T 65576af2c4bb4f25b4d14963cbe65ca7 P a506969c1a5346c09c19a01f6cbc1a8b: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:33.634349  3562 tablet_copy_client.cc:647] T 65576af2c4bb4f25b4d14963cbe65ca7 P a506969c1a5346c09c19a01f6cbc1a8b: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:33.645385  3562 tablet_copy_client.cc:523] T 65576af2c4bb4f25b4d14963cbe65ca7 P a506969c1a5346c09c19a01f6cbc1a8b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:33.652472  3562 tablet_bootstrap.cc:492] T 65576af2c4bb4f25b4d14963cbe65ca7 P a506969c1a5346c09c19a01f6cbc1a8b: Bootstrap starting.
I20240502 01:54:33.891036   497 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:33.900528 31896 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:54:33.949007 31764 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:54:34.097862 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:54:34.210853   630 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:54:34.256178   766 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:34.350706   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:35.690558  3543 tablet_bootstrap.cc:492] T 559610821917470fbcf5641d36acf409 P a506969c1a5346c09c19a01f6cbc1a8b: Bootstrap replayed 1/1 log segments. Stats: ops{read=727 overwritten=0 applied=727 ignored=0} inserts{seen=1397 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:35.693210  3543 tablet_bootstrap.cc:492] T 559610821917470fbcf5641d36acf409 P a506969c1a5346c09c19a01f6cbc1a8b: Bootstrap complete.
I20240502 01:54:35.693779  3543 ts_tablet_manager.cc:1367] T 559610821917470fbcf5641d36acf409 P a506969c1a5346c09c19a01f6cbc1a8b: Time spent bootstrapping tablet: real 2.591s	user 0.853s	sys 0.089s
I20240502 01:54:35.694839  3543 raft_consensus.cc:348] T 559610821917470fbcf5641d36acf409 P a506969c1a5346c09c19a01f6cbc1a8b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 708 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } }
I20240502 01:54:35.704015  3543 raft_consensus.cc:729] T 559610821917470fbcf5641d36acf409 P a506969c1a5346c09c19a01f6cbc1a8b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a506969c1a5346c09c19a01f6cbc1a8b, State: Initialized, Role: FOLLOWER
I20240502 01:54:35.704430  3543 consensus_queue.cc:260] T 559610821917470fbcf5641d36acf409 P a506969c1a5346c09c19a01f6cbc1a8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 727, Last appended: 1.727, Last appended by leader: 727, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 708 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } }
I20240502 01:54:35.708206  3543 ts_tablet_manager.cc:1398] T 559610821917470fbcf5641d36acf409 P a506969c1a5346c09c19a01f6cbc1a8b: Time spent starting tablet: real 0.014s	user 0.002s	sys 0.000s
I20240502 01:54:35.712203 31936 tablet_copy_service.cc:342] P 1864f9ea44c54742b090c594a5be3669: Request end of tablet copy session a506969c1a5346c09c19a01f6cbc1a8b-559610821917470fbcf5641d36acf409 received from {username='slave'} at 127.25.109.133:44189
I20240502 01:54:35.712378 31936 tablet_copy_service.cc:433] P 1864f9ea44c54742b090c594a5be3669: ending tablet copy session a506969c1a5346c09c19a01f6cbc1a8b-559610821917470fbcf5641d36acf409 on tablet 559610821917470fbcf5641d36acf409 with peer a506969c1a5346c09c19a01f6cbc1a8b
W20240502 01:54:35.783363 31719 consensus_peers.cc:485] T 65576af2c4bb4f25b4d14963cbe65ca7 P 35f356f20e274db7bf020f78f1bc9ae7 -> Peer a506969c1a5346c09c19a01f6cbc1a8b (127.25.109.133:46379): Couldn't send request to peer a506969c1a5346c09c19a01f6cbc1a8b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240502 01:54:35.814208   651 raft_consensus.cc:1203] T 559610821917470fbcf5641d36acf409 P a506969c1a5346c09c19a01f6cbc1a8b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.707->[1.708-1.762]   Dedup: 1.727->[1.728-1.762]
W20240502 01:54:35.861232 31719 consensus_peers.cc:485] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 -> Peer a506969c1a5346c09c19a01f6cbc1a8b (127.25.109.133:46379): Couldn't send request to peer a506969c1a5346c09c19a01f6cbc1a8b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240502 01:54:36.445629   497 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:36.481261 31764 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:54:36.562175 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:54:36.606091   766 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:36.611276   630 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:54:36.648623 31896 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:54:36.658824  3550 tablet_bootstrap.cc:492] T 10aa745ed48d46a3b0558d35dee6a807 P a506969c1a5346c09c19a01f6cbc1a8b: Bootstrap replayed 1/1 log segments. Stats: ops{read=777 overwritten=0 applied=777 ignored=0} inserts{seen=1406 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:36.670617  3550 tablet_bootstrap.cc:492] T 10aa745ed48d46a3b0558d35dee6a807 P a506969c1a5346c09c19a01f6cbc1a8b: Bootstrap complete.
I20240502 01:54:36.672152  3550 ts_tablet_manager.cc:1367] T 10aa745ed48d46a3b0558d35dee6a807 P a506969c1a5346c09c19a01f6cbc1a8b: Time spent bootstrapping tablet: real 3.093s	user 0.889s	sys 0.079s
I20240502 01:54:36.677944   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:36.685520  3550 raft_consensus.cc:348] T 10aa745ed48d46a3b0558d35dee6a807 P a506969c1a5346c09c19a01f6cbc1a8b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 762 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } }
I20240502 01:54:36.699967  3550 raft_consensus.cc:729] T 10aa745ed48d46a3b0558d35dee6a807 P a506969c1a5346c09c19a01f6cbc1a8b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a506969c1a5346c09c19a01f6cbc1a8b, State: Initialized, Role: FOLLOWER
I20240502 01:54:36.700381  3550 consensus_queue.cc:260] T 10aa745ed48d46a3b0558d35dee6a807 P a506969c1a5346c09c19a01f6cbc1a8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 777, Last appended: 2.777, Last appended by leader: 777, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 762 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } }
I20240502 01:54:36.701232  3550 ts_tablet_manager.cc:1398] T 10aa745ed48d46a3b0558d35dee6a807 P a506969c1a5346c09c19a01f6cbc1a8b: Time spent starting tablet: real 0.027s	user 0.002s	sys 0.000s
I20240502 01:54:36.702090 31805 tablet_copy_service.cc:342] P 35f356f20e274db7bf020f78f1bc9ae7: Request end of tablet copy session a506969c1a5346c09c19a01f6cbc1a8b-10aa745ed48d46a3b0558d35dee6a807 received from {username='slave'} at 127.25.109.133:60367
I20240502 01:54:36.702208 31805 tablet_copy_service.cc:433] P 35f356f20e274db7bf020f78f1bc9ae7: ending tablet copy session a506969c1a5346c09c19a01f6cbc1a8b-10aa745ed48d46a3b0558d35dee6a807 on tablet 10aa745ed48d46a3b0558d35dee6a807 with peer a506969c1a5346c09c19a01f6cbc1a8b
I20240502 01:54:36.777705  3562 tablet_bootstrap.cc:492] T 65576af2c4bb4f25b4d14963cbe65ca7 P a506969c1a5346c09c19a01f6cbc1a8b: Bootstrap replayed 1/1 log segments. Stats: ops{read=807 overwritten=0 applied=806 ignored=0} inserts{seen=1491 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 01:54:36.778316  3562 tablet_bootstrap.cc:492] T 65576af2c4bb4f25b4d14963cbe65ca7 P a506969c1a5346c09c19a01f6cbc1a8b: Bootstrap complete.
I20240502 01:54:36.778790  3562 ts_tablet_manager.cc:1367] T 65576af2c4bb4f25b4d14963cbe65ca7 P a506969c1a5346c09c19a01f6cbc1a8b: Time spent bootstrapping tablet: real 3.126s	user 0.913s	sys 0.083s
I20240502 01:54:36.779714  3562 raft_consensus.cc:348] T 65576af2c4bb4f25b4d14963cbe65ca7 P a506969c1a5346c09c19a01f6cbc1a8b [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 793 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } }
I20240502 01:54:36.780480  3562 raft_consensus.cc:729] T 65576af2c4bb4f25b4d14963cbe65ca7 P a506969c1a5346c09c19a01f6cbc1a8b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a506969c1a5346c09c19a01f6cbc1a8b, State: Initialized, Role: FOLLOWER
I20240502 01:54:36.780840  3562 consensus_queue.cc:260] T 65576af2c4bb4f25b4d14963cbe65ca7 P a506969c1a5346c09c19a01f6cbc1a8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 806, Last appended: 1.807, Last appended by leader: 807, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 793 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } }
I20240502 01:54:36.781452  3562 ts_tablet_manager.cc:1398] T 65576af2c4bb4f25b4d14963cbe65ca7 P a506969c1a5346c09c19a01f6cbc1a8b: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:54:36.784849 31805 tablet_copy_service.cc:342] P 35f356f20e274db7bf020f78f1bc9ae7: Request end of tablet copy session a506969c1a5346c09c19a01f6cbc1a8b-65576af2c4bb4f25b4d14963cbe65ca7 received from {username='slave'} at 127.25.109.133:60367
I20240502 01:54:36.785001 31805 tablet_copy_service.cc:433] P 35f356f20e274db7bf020f78f1bc9ae7: ending tablet copy session a506969c1a5346c09c19a01f6cbc1a8b-65576af2c4bb4f25b4d14963cbe65ca7 on tablet 65576af2c4bb4f25b4d14963cbe65ca7 with peer a506969c1a5346c09c19a01f6cbc1a8b
I20240502 01:54:36.806026   651 raft_consensus.cc:1203] T 10aa745ed48d46a3b0558d35dee6a807 P a506969c1a5346c09c19a01f6cbc1a8b [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.761->[2.762-2.809]   Dedup: 2.777->[2.778-2.809]
I20240502 01:54:37.135053   652 raft_consensus.cc:1203] T 65576af2c4bb4f25b4d14963cbe65ca7 P a506969c1a5346c09c19a01f6cbc1a8b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.792->[1.793-1.847]   Dedup: 1.807->[1.808-1.847]
I20240502 01:54:37.838016   497 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:37.931747 31896 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:54:37.951449 31764 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:54:38.118789 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:54:38.125106   766 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:38.145529   630 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:54:38.231333   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:38.461490 31783 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "10aa745ed48d46a3b0558d35dee6a807"
dest_uuid: "35f356f20e274db7bf020f78f1bc9ae7"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:45584
I20240502 01:54:38.461695 31783 raft_consensus.cc:595] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [term 2 LEADER]: Received request to transfer leadership
I20240502 01:54:38.534927  3720 raft_consensus.cc:979] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7: : Instructing follower 0aea20ce2a1144519ed144383c016d78 to start an election
I20240502 01:54:38.535195  3720 raft_consensus.cc:1067] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [term 2 LEADER]: Signalling peer 0aea20ce2a1144519ed144383c016d78 to start an election
I20240502 01:54:38.536437   517 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "10aa745ed48d46a3b0558d35dee6a807"
dest_uuid: "0aea20ce2a1144519ed144383c016d78"
 from {username='slave'} at 127.25.109.130:51521
I20240502 01:54:38.536788   517 raft_consensus.cc:482] T 10aa745ed48d46a3b0558d35dee6a807 P 0aea20ce2a1144519ed144383c016d78 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:38.536911   517 raft_consensus.cc:3037] T 10aa745ed48d46a3b0558d35dee6a807 P 0aea20ce2a1144519ed144383c016d78 [term 2 FOLLOWER]: Advancing to term 3
W20240502 01:54:38.541687 31743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47888: Service unavailable: leader transfer in progress
W20240502 01:54:38.549247 31743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47888: Service unavailable: leader transfer in progress
I20240502 01:54:38.539775   517 raft_consensus.cc:504] T 10aa745ed48d46a3b0558d35dee6a807 P 0aea20ce2a1144519ed144383c016d78 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 762 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } }
I20240502 01:54:38.556381   517 leader_election.cc:288] T 10aa745ed48d46a3b0558d35dee6a807 P 0aea20ce2a1144519ed144383c016d78 [CANDIDATE]: Term 3 election: Requested vote from peers 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130:41639), a506969c1a5346c09c19a01f6cbc1a8b (127.25.109.133:46379)
I20240502 01:54:38.557245 31783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10aa745ed48d46a3b0558d35dee6a807" candidate_uuid: "0aea20ce2a1144519ed144383c016d78" candidate_term: 3 candidate_status { last_received { term: 2 index: 841 } } ignore_live_leader: true dest_uuid: "35f356f20e274db7bf020f78f1bc9ae7"
W20240502 01:54:38.557372 31744 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47888: Service unavailable: leader transfer in progress
I20240502 01:54:38.557495 31783 raft_consensus.cc:3032] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:54:38.557591 31783 raft_consensus.cc:729] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 35f356f20e274db7bf020f78f1bc9ae7, State: Running, Role: LEADER
I20240502 01:54:38.557866 31783 consensus_queue.cc:260] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 841, Committed index: 841, Last appended: 2.841, Last appended by leader: 841, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 762 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } }
I20240502 01:54:38.558264 31783 raft_consensus.cc:3037] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:38.562525 31783 raft_consensus.cc:2454] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0aea20ce2a1144519ed144383c016d78 in term 3.
I20240502 01:54:38.564090   449 leader_election.cc:302] T 10aa745ed48d46a3b0558d35dee6a807 P 0aea20ce2a1144519ed144383c016d78 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aea20ce2a1144519ed144383c016d78, 35f356f20e274db7bf020f78f1bc9ae7; no voters: 
W20240502 01:54:38.575014 31744 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47888: Illegal state: replica 35f356f20e274db7bf020f78f1bc9ae7 is not leader of this config: current role FOLLOWER
I20240502 01:54:38.577231  3784 raft_consensus.cc:2781] T 10aa745ed48d46a3b0558d35dee6a807 P 0aea20ce2a1144519ed144383c016d78 [term 3 FOLLOWER]: Leader election won for term 3
W20240502 01:54:38.604516 31743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47888: Illegal state: replica 35f356f20e274db7bf020f78f1bc9ae7 is not leader of this config: current role FOLLOWER
I20240502 01:54:38.616297  3784 raft_consensus.cc:686] T 10aa745ed48d46a3b0558d35dee6a807 P 0aea20ce2a1144519ed144383c016d78 [term 3 LEADER]: Becoming Leader. State: Replica: 0aea20ce2a1144519ed144383c016d78, State: Running, Role: LEADER
I20240502 01:54:38.617466  3784 consensus_queue.cc:237] T 10aa745ed48d46a3b0558d35dee6a807 P 0aea20ce2a1144519ed144383c016d78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 841, Committed index: 841, Last appended: 2.841, Last appended by leader: 841, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 762 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } }
I20240502 01:54:38.625814 31502 catalog_manager.cc:5484] T 10aa745ed48d46a3b0558d35dee6a807 P 0aea20ce2a1144519ed144383c016d78 reported cstate change: term changed from 2 to 3, leader changed from 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130) to 0aea20ce2a1144519ed144383c016d78 (127.25.109.132). New cstate: current_term: 3 leader_uuid: "0aea20ce2a1144519ed144383c016d78" committed_config { opid_index: 762 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } }
I20240502 01:54:38.626345   651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10aa745ed48d46a3b0558d35dee6a807" candidate_uuid: "0aea20ce2a1144519ed144383c016d78" candidate_term: 3 candidate_status { last_received { term: 2 index: 841 } } ignore_live_leader: true dest_uuid: "a506969c1a5346c09c19a01f6cbc1a8b"
I20240502 01:54:38.626669   651 raft_consensus.cc:3037] T 10aa745ed48d46a3b0558d35dee6a807 P a506969c1a5346c09c19a01f6cbc1a8b [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:38.629562   651 raft_consensus.cc:2454] T 10aa745ed48d46a3b0558d35dee6a807 P a506969c1a5346c09c19a01f6cbc1a8b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0aea20ce2a1144519ed144383c016d78 in term 3.
I20240502 01:54:39.025728   652 raft_consensus.cc:1261] T 10aa745ed48d46a3b0558d35dee6a807 P a506969c1a5346c09c19a01f6cbc1a8b [term 3 FOLLOWER]: Refusing update from remote peer 0aea20ce2a1144519ed144383c016d78: Log matching property violated. Preceding OpId in replica: term: 2 index: 841. Preceding OpId from leader: term: 3 index: 843. (index mismatch)
I20240502 01:54:39.032681  3784 consensus_queue.cc:1035] T 10aa745ed48d46a3b0558d35dee6a807 P 0aea20ce2a1144519ed144383c016d78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 842, Last known committed idx: 841, Time since last communication: 0.000s
I20240502 01:54:39.036445 31783 raft_consensus.cc:1261] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [term 3 FOLLOWER]: Refusing update from remote peer 0aea20ce2a1144519ed144383c016d78: Log matching property violated. Preceding OpId in replica: term: 2 index: 841. Preceding OpId from leader: term: 3 index: 843. (index mismatch)
I20240502 01:54:39.038149  3785 consensus_queue.cc:1035] T 10aa745ed48d46a3b0558d35dee6a807 P 0aea20ce2a1144519ed144383c016d78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 842, Last known committed idx: 841, Time since last communication: 0.000s
I20240502 01:54:39.047927  3585 mvcc.cc:205] Tried to move back new op lower bound from 7023062544480960512 to 7023062542828949504. Current Snapshot: MvccSnapshot[applied={T|T < 7023062544480960512}]
I20240502 01:54:39.668195   497 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:39.744540   766 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:39.760222 31764 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:54:39.809954   630 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 01:54:39.870759 31850 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.129:42243, user_credentials={real_user=slave}} blocked reactor thread for 65167.7us
I20240502 01:54:39.917502 31896 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:54:39.925117 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:54:39.995359   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:40.301834   516 consensus_queue.cc:237] T 10aa745ed48d46a3b0558d35dee6a807 P 0aea20ce2a1144519ed144383c016d78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 860, Committed index: 860, Last appended: 3.860, Last appended by leader: 841, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 861 OBSOLETE_local: false peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } }
I20240502 01:54:40.315178   651 raft_consensus.cc:1261] T 10aa745ed48d46a3b0558d35dee6a807 P a506969c1a5346c09c19a01f6cbc1a8b [term 3 FOLLOWER]: Refusing update from remote peer 0aea20ce2a1144519ed144383c016d78: Log matching property violated. Preceding OpId in replica: term: 3 index: 860. Preceding OpId from leader: term: 3 index: 861. (index mismatch)
I20240502 01:54:40.316490  3877 consensus_queue.cc:1035] T 10aa745ed48d46a3b0558d35dee6a807 P 0aea20ce2a1144519ed144383c016d78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 861, Last known committed idx: 860, Time since last communication: 0.000s
I20240502 01:54:40.321055  3784 raft_consensus.cc:2932] T 10aa745ed48d46a3b0558d35dee6a807 P 0aea20ce2a1144519ed144383c016d78 [term 3 LEADER]: Committing config change with OpId 3.861: config changed from index 762 to 861, VOTER 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130) evicted. New config: { opid_index: 861 OBSOLETE_local: false peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } }
I20240502 01:54:40.341006 31501 catalog_manager.cc:5484] T 10aa745ed48d46a3b0558d35dee6a807 P 0aea20ce2a1144519ed144383c016d78 reported cstate change: config changed from index 762 to 861, VOTER 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130) evicted. New cstate: current_term: 3 leader_uuid: "0aea20ce2a1144519ed144383c016d78" committed_config { opid_index: 861 OBSOLETE_local: false peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } }
I20240502 01:54:40.341544   651 raft_consensus.cc:2932] T 10aa745ed48d46a3b0558d35dee6a807 P a506969c1a5346c09c19a01f6cbc1a8b [term 3 FOLLOWER]: Committing config change with OpId 3.861: config changed from index 762 to 861, VOTER 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130) evicted. New config: { opid_index: 861 OBSOLETE_local: false peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } }
I20240502 01:54:40.444280 31764 tablet_service.cc:1514] Processing DeleteTablet for tablet 10aa745ed48d46a3b0558d35dee6a807 with delete_type TABLET_DATA_TOMBSTONED (TS 35f356f20e274db7bf020f78f1bc9ae7 not found in new config with opid_index 861) from {username='slave'} at 127.0.0.1:46798
I20240502 01:54:40.444999  3894 tablet_replica.cc:330] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7: stopping tablet replica
I20240502 01:54:40.445351  3894 raft_consensus.cc:2229] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:40.445642  3894 pending_rounds.cc:62] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7: Trying to abort 1 pending ops.
I20240502 01:54:40.445755  3894 pending_rounds.cc:68] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7: Aborting op as it isn't in flight: id { term: 3 index: 860 } timestamp: 7023062549623062528 op_type: WRITE_OP write_request { tablet_id: "10aa745ed48d46a3b0558d35dee6a807" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\254\252\214m\340\222Hz\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""3377607044b8e6eb" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023062549543555072 authz_token { token_data: "\010\317\346\313\261\006\"3\n\005slave\022*\n c997c25d61cd4f25aad452793cf51df8\020\001\030\001 \001(\001" signature: "<redacted>""\214l\'\255j{\021\337U\016\010v\217_j\202:\014\233\304\374\301g\323W\346\361\274\326\373\230\035QF \210nN\316^\376\224\336 \226E\036\335\303\206]\323\274\016\371ny]6\332\233o\235\256" signing_key_seq_num: 0 } } request_id { client_id: "8a9503aff5064d998087c2ef1e059837" seq_no: 2614 first_incomplete_seq_no: 2614 attempt_no: 0 }
I20240502 01:54:40.446449  3894 raft_consensus.cc:2258] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:40.448263  3894 ts_tablet_manager.cc:1875] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:40.456127  3894 ts_tablet_manager.cc:1888] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.860
I20240502 01:54:40.456393  3894 log.cc:1196] T 10aa745ed48d46a3b0558d35dee6a807 P 35f356f20e274db7bf020f78f1bc9ae7: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal/wals/10aa745ed48d46a3b0558d35dee6a807
I20240502 01:54:40.457612 31490 catalog_manager.cc:4830] TS 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130:41639): tablet 10aa745ed48d46a3b0558d35dee6a807 (table rebalance_test_table_0 [id=c997c25d61cd4f25aad452793cf51df8]) successfully deleted
W20240502 01:54:40.873500 31719 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.133:46379, user_credentials={real_user=slave}} blocked reactor thread for 56249.5us
I20240502 01:54:41.137517   497 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:41.216470 31896 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:54:41.241012   630 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:54:41.293470   766 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:41.315446 31764 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:54:41.414175 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:54:41.459686   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:41.870396 31912 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "559610821917470fbcf5641d36acf409"
dest_uuid: "1864f9ea44c54742b090c594a5be3669"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:43460
I20240502 01:54:41.870616 31912 raft_consensus.cc:595] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669 [term 1 LEADER]: Received request to transfer leadership
W20240502 01:54:42.006587 31875 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38400: Service unavailable: leader transfer in progress
W20240502 01:54:42.016918 31875 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38400: Service unavailable: leader transfer in progress
W20240502 01:54:42.029876 31875 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38400: Service unavailable: leader transfer in progress
W20240502 01:54:42.039080 31875 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38400: Service unavailable: leader transfer in progress
W20240502 01:54:42.049372 31876 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38400: Service unavailable: leader transfer in progress
W20240502 01:54:42.064769 31875 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38400: Service unavailable: leader transfer in progress
W20240502 01:54:42.085991 31875 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38400: Service unavailable: leader transfer in progress
W20240502 01:54:42.105224 31875 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38400: Service unavailable: leader transfer in progress
W20240502 01:54:42.134042 31875 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38400: Service unavailable: leader transfer in progress
W20240502 01:54:42.149183 31875 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38400: Service unavailable: leader transfer in progress
W20240502 01:54:42.169121 31875 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38400: Service unavailable: leader transfer in progress
W20240502 01:54:42.184271 31875 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38400: Service unavailable: leader transfer in progress
W20240502 01:54:42.201946 31875 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38400: Service unavailable: leader transfer in progress
W20240502 01:54:42.217608 31875 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38400: Service unavailable: leader transfer in progress
W20240502 01:54:42.237041 31875 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38400: Service unavailable: leader transfer in progress
W20240502 01:54:42.256650 31875 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38400: Service unavailable: leader transfer in progress
W20240502 01:54:42.279381 31875 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38400: Service unavailable: leader transfer in progress
W20240502 01:54:42.317971 31875 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38400: Service unavailable: leader transfer in progress
W20240502 01:54:42.343375 31875 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38400: Service unavailable: leader transfer in progress
W20240502 01:54:42.366544 31876 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38400: Service unavailable: leader transfer in progress
I20240502 01:54:42.390595  3896 raft_consensus.cc:979] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669: : Instructing follower a506969c1a5346c09c19a01f6cbc1a8b to start an election
I20240502 01:54:42.390786  3896 raft_consensus.cc:1067] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669 [term 1 LEADER]: Signalling peer a506969c1a5346c09c19a01f6cbc1a8b to start an election
W20240502 01:54:42.391528 31875 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38400: Service unavailable: leader transfer in progress
I20240502 01:54:42.396404   649 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "559610821917470fbcf5641d36acf409"
dest_uuid: "a506969c1a5346c09c19a01f6cbc1a8b"
 from {username='slave'} at 127.25.109.131:48649
I20240502 01:54:42.396668   649 raft_consensus.cc:482] T 559610821917470fbcf5641d36acf409 P a506969c1a5346c09c19a01f6cbc1a8b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:42.396828   649 raft_consensus.cc:3037] T 559610821917470fbcf5641d36acf409 P a506969c1a5346c09c19a01f6cbc1a8b [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:42.400034   649 raft_consensus.cc:504] T 559610821917470fbcf5641d36acf409 P a506969c1a5346c09c19a01f6cbc1a8b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 708 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } }
I20240502 01:54:42.401940 31912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "559610821917470fbcf5641d36acf409" candidate_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" candidate_term: 2 candidate_status { last_received { term: 1 index: 830 } } ignore_live_leader: true dest_uuid: "1864f9ea44c54742b090c594a5be3669"
I20240502 01:54:42.402185 31912 raft_consensus.cc:3032] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:42.402289 31912 raft_consensus.cc:729] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1864f9ea44c54742b090c594a5be3669, State: Running, Role: LEADER
I20240502 01:54:42.402568 31912 consensus_queue.cc:260] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 830, Committed index: 830, Last appended: 1.830, Last appended by leader: 830, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 708 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } }
I20240502 01:54:42.402923 31912 raft_consensus.cc:3037] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:42.410372 31912 raft_consensus.cc:2454] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a506969c1a5346c09c19a01f6cbc1a8b in term 2.
I20240502 01:54:42.415688   649 leader_election.cc:288] T 559610821917470fbcf5641d36acf409 P a506969c1a5346c09c19a01f6cbc1a8b [CANDIDATE]: Term 2 election: Requested vote from peers 1864f9ea44c54742b090c594a5be3669 (127.25.109.131:42119), facf1046c8054d2d841cf875c63e29e2 (127.25.109.135:37253)
I20240502 01:54:42.430804   917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "559610821917470fbcf5641d36acf409" candidate_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" candidate_term: 2 candidate_status { last_received { term: 1 index: 830 } } ignore_live_leader: true dest_uuid: "facf1046c8054d2d841cf875c63e29e2"
I20240502 01:54:42.431244   917 raft_consensus.cc:3037] T 559610821917470fbcf5641d36acf409 P facf1046c8054d2d841cf875c63e29e2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:42.434723   917 raft_consensus.cc:2454] T 559610821917470fbcf5641d36acf409 P facf1046c8054d2d841cf875c63e29e2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a506969c1a5346c09c19a01f6cbc1a8b in term 2.
I20240502 01:54:42.435338   584 leader_election.cc:302] T 559610821917470fbcf5641d36acf409 P a506969c1a5346c09c19a01f6cbc1a8b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a506969c1a5346c09c19a01f6cbc1a8b, facf1046c8054d2d841cf875c63e29e2; no voters: 
I20240502 01:54:42.435726  3836 raft_consensus.cc:2781] T 559610821917470fbcf5641d36acf409 P a506969c1a5346c09c19a01f6cbc1a8b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:42.435945  3836 raft_consensus.cc:686] T 559610821917470fbcf5641d36acf409 P a506969c1a5346c09c19a01f6cbc1a8b [term 2 LEADER]: Becoming Leader. State: Replica: a506969c1a5346c09c19a01f6cbc1a8b, State: Running, Role: LEADER
I20240502 01:54:42.436300  3836 consensus_queue.cc:237] T 559610821917470fbcf5641d36acf409 P a506969c1a5346c09c19a01f6cbc1a8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 830, Committed index: 830, Last appended: 1.830, Last appended by leader: 830, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 708 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } }
I20240502 01:54:42.440042 31502 catalog_manager.cc:5484] T 559610821917470fbcf5641d36acf409 P a506969c1a5346c09c19a01f6cbc1a8b reported cstate change: term changed from 1 to 2, leader changed from 1864f9ea44c54742b090c594a5be3669 (127.25.109.131) to a506969c1a5346c09c19a01f6cbc1a8b (127.25.109.133). New cstate: current_term: 2 leader_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" committed_config { opid_index: 708 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } }
W20240502 01:54:42.421909 31875 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38400: Illegal state: replica 1864f9ea44c54742b090c594a5be3669 is not leader of this config: current role FOLLOWER
I20240502 01:54:42.607442   919 raft_consensus.cc:1261] T 559610821917470fbcf5641d36acf409 P facf1046c8054d2d841cf875c63e29e2 [term 2 FOLLOWER]: Refusing update from remote peer a506969c1a5346c09c19a01f6cbc1a8b: Log matching property violated. Preceding OpId in replica: term: 1 index: 830. Preceding OpId from leader: term: 2 index: 832. (index mismatch)
I20240502 01:54:42.608904 31912 raft_consensus.cc:1261] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669 [term 2 FOLLOWER]: Refusing update from remote peer a506969c1a5346c09c19a01f6cbc1a8b: Log matching property violated. Preceding OpId in replica: term: 1 index: 830. Preceding OpId from leader: term: 2 index: 832. (index mismatch)
I20240502 01:54:42.610661  3157 consensus_queue.cc:1035] T 559610821917470fbcf5641d36acf409 P a506969c1a5346c09c19a01f6cbc1a8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 831, Last known committed idx: 830, Time since last communication: 0.000s
I20240502 01:54:42.612062  3157 consensus_queue.cc:1035] T 559610821917470fbcf5641d36acf409 P a506969c1a5346c09c19a01f6cbc1a8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 831, Last known committed idx: 830, Time since last communication: 0.000s
I20240502 01:54:42.745966  3953 mvcc.cc:205] Tried to move back new op lower bound from 7023062559150944256 to 7023062558462128128. Current Snapshot: MvccSnapshot[applied={T|T < 7023062559150944256 or (T in {7023062559150944256})}]
I20240502 01:54:42.814711   497 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:54:42.847852 31852 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.135:37253, user_credentials={real_user=slave}} blocked reactor thread for 52740.3us
I20240502 01:54:42.883656 31896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 01:54:42.886579 31719 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.135:37253, user_credentials={real_user=slave}} blocked reactor thread for 55423.5us
I20240502 01:54:42.912513   630 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:54:42.929471 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:54:42.977008 31764 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:54:42.977732   766 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:43.020152   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:43.825757   651 consensus_queue.cc:237] T 559610821917470fbcf5641d36acf409 P a506969c1a5346c09c19a01f6cbc1a8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 854, Committed index: 854, Last appended: 2.854, Last appended by leader: 830, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 855 OBSOLETE_local: false peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } }
I20240502 01:54:43.828231   915 raft_consensus.cc:1261] T 559610821917470fbcf5641d36acf409 P facf1046c8054d2d841cf875c63e29e2 [term 2 FOLLOWER]: Refusing update from remote peer a506969c1a5346c09c19a01f6cbc1a8b: Log matching property violated. Preceding OpId in replica: term: 2 index: 854. Preceding OpId from leader: term: 2 index: 855. (index mismatch)
I20240502 01:54:43.829365  4055 consensus_queue.cc:1035] T 559610821917470fbcf5641d36acf409 P a506969c1a5346c09c19a01f6cbc1a8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 855, Last known committed idx: 854, Time since last communication: 0.000s
I20240502 01:54:43.833833  4055 raft_consensus.cc:2932] T 559610821917470fbcf5641d36acf409 P a506969c1a5346c09c19a01f6cbc1a8b [term 2 LEADER]: Committing config change with OpId 2.855: config changed from index 708 to 855, VOTER 1864f9ea44c54742b090c594a5be3669 (127.25.109.131) evicted. New config: { opid_index: 855 OBSOLETE_local: false peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } }
I20240502 01:54:43.837615   920 raft_consensus.cc:2932] T 559610821917470fbcf5641d36acf409 P facf1046c8054d2d841cf875c63e29e2 [term 2 FOLLOWER]: Committing config change with OpId 2.855: config changed from index 708 to 855, VOTER 1864f9ea44c54742b090c594a5be3669 (127.25.109.131) evicted. New config: { opid_index: 855 OBSOLETE_local: false peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } }
I20240502 01:54:43.850929 31502 catalog_manager.cc:5484] T 559610821917470fbcf5641d36acf409 P a506969c1a5346c09c19a01f6cbc1a8b reported cstate change: config changed from index 708 to 855, VOTER 1864f9ea44c54742b090c594a5be3669 (127.25.109.131) evicted. New cstate: current_term: 2 leader_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" committed_config { opid_index: 855 OBSOLETE_local: false peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } }
I20240502 01:54:44.145016 31896 tablet_service.cc:1514] Processing DeleteTablet for tablet 559610821917470fbcf5641d36acf409 with delete_type TABLET_DATA_TOMBSTONED (TS 1864f9ea44c54742b090c594a5be3669 not found in new config with opid_index 855) from {username='slave'} at 127.0.0.1:40670
I20240502 01:54:44.230274  4074 tablet_replica.cc:330] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669: stopping tablet replica
I20240502 01:54:44.230684  4074 raft_consensus.cc:2229] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:44.231014  4074 raft_consensus.cc:2258] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:44.240648  4074 ts_tablet_manager.cc:1875] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:44.259142  4074 ts_tablet_manager.cc:1888] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.854
I20240502 01:54:44.259429  4074 log.cc:1196] T 559610821917470fbcf5641d36acf409 P 1864f9ea44c54742b090c594a5be3669: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/wals/559610821917470fbcf5641d36acf409
I20240502 01:54:44.260659 31489 catalog_manager.cc:4830] TS 1864f9ea44c54742b090c594a5be3669 (127.25.109.131:42119): tablet 559610821917470fbcf5641d36acf409 (table rebalance_test_table_1 [id=76d2ee30f2bb480087daf717aa077ba2]) successfully deleted
I20240502 01:54:45.078132 31764 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:54:45.130406 31896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:54:45.244851   497 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:45.293030 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:54:45.308184   766 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:45.326054   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:45.340977   630 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:54:45.849046 31783 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "65576af2c4bb4f25b4d14963cbe65ca7"
dest_uuid: "35f356f20e274db7bf020f78f1bc9ae7"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:56078
I20240502 01:54:45.849272 31783 raft_consensus.cc:595] T 65576af2c4bb4f25b4d14963cbe65ca7 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 LEADER]: Received request to transfer leadership
W20240502 01:54:45.899912 31744 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47902: Service unavailable: leader transfer in progress
W20240502 01:54:45.902719 31742 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47902: Service unavailable: leader transfer in progress
W20240502 01:54:45.908723 31742 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47902: Service unavailable: leader transfer in progress
W20240502 01:54:45.920622 31742 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47902: Service unavailable: leader transfer in progress
W20240502 01:54:45.927907 31742 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47902: Service unavailable: leader transfer in progress
W20240502 01:54:45.937176 31743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47902: Service unavailable: leader transfer in progress
W20240502 01:54:45.946956 31743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47902: Service unavailable: leader transfer in progress
W20240502 01:54:45.969259 31743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47902: Service unavailable: leader transfer in progress
W20240502 01:54:45.983697 31743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47902: Service unavailable: leader transfer in progress
W20240502 01:54:45.998446 31743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47902: Service unavailable: leader transfer in progress
W20240502 01:54:46.013758 31743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47902: Service unavailable: leader transfer in progress
W20240502 01:54:46.031877 31743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47902: Service unavailable: leader transfer in progress
W20240502 01:54:46.055817 31744 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47902: Service unavailable: leader transfer in progress
W20240502 01:54:46.072875 31743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47902: Service unavailable: leader transfer in progress
W20240502 01:54:46.092041 31743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47902: Service unavailable: leader transfer in progress
W20240502 01:54:46.109472 31742 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47902: Service unavailable: leader transfer in progress
W20240502 01:54:46.132208 31742 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47902: Service unavailable: leader transfer in progress
W20240502 01:54:46.154394 31742 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47902: Service unavailable: leader transfer in progress
W20240502 01:54:46.182216 31742 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47902: Service unavailable: leader transfer in progress
W20240502 01:54:46.206284 31742 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47902: Service unavailable: leader transfer in progress
W20240502 01:54:46.243036 31742 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47902: Service unavailable: leader transfer in progress
W20240502 01:54:46.268393 31742 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47902: Service unavailable: leader transfer in progress
W20240502 01:54:46.295991 31742 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47902: Service unavailable: leader transfer in progress
I20240502 01:54:46.310621  4137 raft_consensus.cc:979] T 65576af2c4bb4f25b4d14963cbe65ca7 P 35f356f20e274db7bf020f78f1bc9ae7: : Instructing follower a506969c1a5346c09c19a01f6cbc1a8b to start an election
I20240502 01:54:46.310838  4137 raft_consensus.cc:1067] T 65576af2c4bb4f25b4d14963cbe65ca7 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 LEADER]: Signalling peer a506969c1a5346c09c19a01f6cbc1a8b to start an election
I20240502 01:54:46.311703   651 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "65576af2c4bb4f25b4d14963cbe65ca7"
dest_uuid: "a506969c1a5346c09c19a01f6cbc1a8b"
 from {username='slave'} at 127.25.109.130:35939
I20240502 01:54:46.311965   651 raft_consensus.cc:482] T 65576af2c4bb4f25b4d14963cbe65ca7 P a506969c1a5346c09c19a01f6cbc1a8b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:46.312108   651 raft_consensus.cc:3037] T 65576af2c4bb4f25b4d14963cbe65ca7 P a506969c1a5346c09c19a01f6cbc1a8b [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:46.314864   651 raft_consensus.cc:504] T 65576af2c4bb4f25b4d14963cbe65ca7 P a506969c1a5346c09c19a01f6cbc1a8b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 793 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } }
I20240502 01:54:46.316251 31913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65576af2c4bb4f25b4d14963cbe65ca7" candidate_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" candidate_term: 2 candidate_status { last_received { term: 1 index: 994 } } ignore_live_leader: true dest_uuid: "1864f9ea44c54742b090c594a5be3669"
I20240502 01:54:46.316494 31913 raft_consensus.cc:3037] T 65576af2c4bb4f25b4d14963cbe65ca7 P 1864f9ea44c54742b090c594a5be3669 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:46.316932   651 leader_election.cc:288] T 65576af2c4bb4f25b4d14963cbe65ca7 P a506969c1a5346c09c19a01f6cbc1a8b [CANDIDATE]: Term 2 election: Requested vote from peers 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130:41639), 1864f9ea44c54742b090c594a5be3669 (127.25.109.131:42119)
I20240502 01:54:46.317930 31783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65576af2c4bb4f25b4d14963cbe65ca7" candidate_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" candidate_term: 2 candidate_status { last_received { term: 1 index: 994 } } ignore_live_leader: true dest_uuid: "35f356f20e274db7bf020f78f1bc9ae7"
I20240502 01:54:46.318181 31783 raft_consensus.cc:3032] T 65576af2c4bb4f25b4d14963cbe65ca7 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:46.318276 31783 raft_consensus.cc:729] T 65576af2c4bb4f25b4d14963cbe65ca7 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 35f356f20e274db7bf020f78f1bc9ae7, State: Running, Role: LEADER
I20240502 01:54:46.318567 31783 consensus_queue.cc:260] T 65576af2c4bb4f25b4d14963cbe65ca7 P 35f356f20e274db7bf020f78f1bc9ae7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 994, Committed index: 994, Last appended: 1.994, Last appended by leader: 994, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 793 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } }
I20240502 01:54:46.318964 31783 raft_consensus.cc:3037] T 65576af2c4bb4f25b4d14963cbe65ca7 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:46.319401 31913 raft_consensus.cc:2454] T 65576af2c4bb4f25b4d14963cbe65ca7 P 1864f9ea44c54742b090c594a5be3669 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a506969c1a5346c09c19a01f6cbc1a8b in term 2.
I20240502 01:54:46.320439   581 leader_election.cc:302] T 65576af2c4bb4f25b4d14963cbe65ca7 P a506969c1a5346c09c19a01f6cbc1a8b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1864f9ea44c54742b090c594a5be3669, a506969c1a5346c09c19a01f6cbc1a8b; no voters: 
I20240502 01:54:46.320822  4129 raft_consensus.cc:2781] T 65576af2c4bb4f25b4d14963cbe65ca7 P a506969c1a5346c09c19a01f6cbc1a8b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:46.320988  4129 raft_consensus.cc:686] T 65576af2c4bb4f25b4d14963cbe65ca7 P a506969c1a5346c09c19a01f6cbc1a8b [term 2 LEADER]: Becoming Leader. State: Replica: a506969c1a5346c09c19a01f6cbc1a8b, State: Running, Role: LEADER
I20240502 01:54:46.321306  4129 consensus_queue.cc:237] T 65576af2c4bb4f25b4d14963cbe65ca7 P a506969c1a5346c09c19a01f6cbc1a8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 994, Committed index: 994, Last appended: 1.994, Last appended by leader: 994, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 793 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } }
I20240502 01:54:46.324294 31783 raft_consensus.cc:2454] T 65576af2c4bb4f25b4d14963cbe65ca7 P 35f356f20e274db7bf020f78f1bc9ae7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a506969c1a5346c09c19a01f6cbc1a8b in term 2.
I20240502 01:54:46.325790 31502 catalog_manager.cc:5484] T 65576af2c4bb4f25b4d14963cbe65ca7 P a506969c1a5346c09c19a01f6cbc1a8b reported cstate change: term changed from 1 to 2, leader changed from 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130) to a506969c1a5346c09c19a01f6cbc1a8b (127.25.109.133). New cstate: current_term: 2 leader_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" committed_config { opid_index: 793 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } }
W20240502 01:54:46.328895 31742 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47902: Illegal state: replica 35f356f20e274db7bf020f78f1bc9ae7 is not leader of this config: current role FOLLOWER
W20240502 01:54:46.529731 31876 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34790: Illegal state: replica 1864f9ea44c54742b090c594a5be3669 is not leader of this config: current role FOLLOWER
W20240502 01:54:46.613864 31742 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47902: Illegal state: replica 35f356f20e274db7bf020f78f1bc9ae7 is not leader of this config: current role FOLLOWER
W20240502 01:54:46.644795 31876 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:34790: Illegal state: replica 1864f9ea44c54742b090c594a5be3669 is not leader of this config: current role FOLLOWER
I20240502 01:54:46.712939 31913 raft_consensus.cc:1261] T 65576af2c4bb4f25b4d14963cbe65ca7 P 1864f9ea44c54742b090c594a5be3669 [term 2 FOLLOWER]: Refusing update from remote peer a506969c1a5346c09c19a01f6cbc1a8b: Log matching property violated. Preceding OpId in replica: term: 1 index: 994. Preceding OpId from leader: term: 2 index: 996. (index mismatch)
I20240502 01:54:46.713099 31783 raft_consensus.cc:1261] T 65576af2c4bb4f25b4d14963cbe65ca7 P 35f356f20e274db7bf020f78f1bc9ae7 [term 2 FOLLOWER]: Refusing update from remote peer a506969c1a5346c09c19a01f6cbc1a8b: Log matching property violated. Preceding OpId in replica: term: 1 index: 994. Preceding OpId from leader: term: 2 index: 996. (index mismatch)
I20240502 01:54:46.713635  4129 consensus_queue.cc:1035] T 65576af2c4bb4f25b4d14963cbe65ca7 P a506969c1a5346c09c19a01f6cbc1a8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 995, Last known committed idx: 994, Time since last communication: 0.000s
I20240502 01:54:46.714185  4129 consensus_queue.cc:1035] T 65576af2c4bb4f25b4d14963cbe65ca7 P a506969c1a5346c09c19a01f6cbc1a8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 995, Last known committed idx: 994, Time since last communication: 0.000s
I20240502 01:54:46.955545   497 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:54:47.010823 31584 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.135:37253, user_credentials={real_user=slave}} blocked reactor thread for 79643.1us
I20240502 01:54:47.040665 31896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:54:47.057642 31764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:54:47.097831   766 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:47.117605   630 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:54:47.154049 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:54:47.326638   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:48.240487 31896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:54:48.241724   497 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:48.367200   630 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:54:48.390179   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:48.518265   766 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:48.523414 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:54:48.646636 31764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:54:49.307687   649 consensus_queue.cc:237] T 65576af2c4bb4f25b4d14963cbe65ca7 P a506969c1a5346c09c19a01f6cbc1a8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1043, Committed index: 1043, Last appended: 2.1043, Last appended by leader: 994, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1044 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } }
I20240502 01:54:49.309758 31912 raft_consensus.cc:1261] T 65576af2c4bb4f25b4d14963cbe65ca7 P 1864f9ea44c54742b090c594a5be3669 [term 2 FOLLOWER]: Refusing update from remote peer a506969c1a5346c09c19a01f6cbc1a8b: Log matching property violated. Preceding OpId in replica: term: 2 index: 1043. Preceding OpId from leader: term: 2 index: 1044. (index mismatch)
I20240502 01:54:49.310405  4191 consensus_queue.cc:1035] T 65576af2c4bb4f25b4d14963cbe65ca7 P a506969c1a5346c09c19a01f6cbc1a8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1044, Last known committed idx: 1043, Time since last communication: 0.000s
I20240502 01:54:49.321568  4191 raft_consensus.cc:2932] T 65576af2c4bb4f25b4d14963cbe65ca7 P a506969c1a5346c09c19a01f6cbc1a8b [term 2 LEADER]: Committing config change with OpId 2.1044: config changed from index 793 to 1044, VOTER 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130) evicted. New config: { opid_index: 1044 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } }
I20240502 01:54:49.329995 31912 raft_consensus.cc:2932] T 65576af2c4bb4f25b4d14963cbe65ca7 P 1864f9ea44c54742b090c594a5be3669 [term 2 FOLLOWER]: Committing config change with OpId 2.1044: config changed from index 793 to 1044, VOTER 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130) evicted. New config: { opid_index: 1044 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } }
I20240502 01:54:49.427340 31499 catalog_manager.cc:5484] T 65576af2c4bb4f25b4d14963cbe65ca7 P a506969c1a5346c09c19a01f6cbc1a8b reported cstate change: config changed from index 793 to 1044, VOTER 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130) evicted. New cstate: current_term: 2 leader_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" committed_config { opid_index: 1044 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } }
I20240502 01:54:49.497437 31764 tablet_service.cc:1514] Processing DeleteTablet for tablet 65576af2c4bb4f25b4d14963cbe65ca7 with delete_type TABLET_DATA_TOMBSTONED (TS 35f356f20e274db7bf020f78f1bc9ae7 not found in new config with opid_index 1044) from {username='slave'} at 127.0.0.1:46798
I20240502 01:54:49.502014  4327 tablet_replica.cc:330] T 65576af2c4bb4f25b4d14963cbe65ca7 P 35f356f20e274db7bf020f78f1bc9ae7: stopping tablet replica
I20240502 01:54:49.502400  4327 raft_consensus.cc:2229] T 65576af2c4bb4f25b4d14963cbe65ca7 P 35f356f20e274db7bf020f78f1bc9ae7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:49.502960  4327 raft_consensus.cc:2258] T 65576af2c4bb4f25b4d14963cbe65ca7 P 35f356f20e274db7bf020f78f1bc9ae7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:49.519203  4327 ts_tablet_manager.cc:1875] T 65576af2c4bb4f25b4d14963cbe65ca7 P 35f356f20e274db7bf020f78f1bc9ae7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:49.529101  4327 ts_tablet_manager.cc:1888] T 65576af2c4bb4f25b4d14963cbe65ca7 P 35f356f20e274db7bf020f78f1bc9ae7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.1043
I20240502 01:54:49.529376  4327 log.cc:1196] T 65576af2c4bb4f25b4d14963cbe65ca7 P 35f356f20e274db7bf020f78f1bc9ae7: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal/wals/65576af2c4bb4f25b4d14963cbe65ca7
I20240502 01:54:49.531916 31490 catalog_manager.cc:4830] TS 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130:41639): tablet 65576af2c4bb4f25b4d14963cbe65ca7 (table rebalance_test_table_2 [id=7340ad22b7574a6d81f7eaa977a06414]) successfully deleted
I20240502 01:54:50.536505 31764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:54:50.603394   497 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:50.621142   766 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:50.638314   630 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:54:50.668861 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:54:50.726992 31896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:54:50.785285   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:51.624239 31896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:54:51.640844 31764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:54:51.657930   497 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:51.776716   766 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:51.797108   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:51.836887   630 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:54:51.838233 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:54:54.436053 31764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:54:54.475736   497 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:54.490998 31896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:54:54.502563 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:54:54.567162   630 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:54:54.573534   766 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:54.620064   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:56.266271 31896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:54:56.304138   497 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:56.411388   766 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:56.443217   630 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:54:56.520288 31764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:54:56.543682 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:54:56.608786   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:57.224419 31896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:54:57.226006   497 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:57.244362 31764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:54:57.244762   766 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:57.257439 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:54:57.284396   630 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:54:57.286837   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 15725d2cffce4f5da0ed6c7449588f85 | 127.25.109.190:41417 | HEALTHY

Unusual flags for Master:
                   Flag                   |                                                                                Value                                                                                 |         Tags          |         Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 allow_unsafe_replication_factor          | true                                                                                                                                                                 | runtime,unsafe        | all 1 server(s) checked
 ipki_ca_key_size                         | 768                                                                                                                                                                  | experimental          | all 1 server(s) checked
 ipki_server_key_size                     | 768                                                                                                                                                                  | experimental          | all 1 server(s) checked
 never_fsync                              | true                                                                                                                                                                 | unsafe,advanced       | all 1 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                    | unsafe,hidden         | all 1 server(s) checked
 raft_prepare_replacement_before_eviction | false                                                                                                                                                                | experimental,advanced | all 1 server(s) checked
 rpc_reuseport                            | true                                                                                                                                                                 | experimental          | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                 | unsafe                | all 1 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                   | hidden                | all 1 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden                | all 1 server(s) checked
 tsk_num_rsa_bits                         | 512                                                                                                                                                                  | experimental          | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |        Value         |         Master
---------------------+----------------------+-------------------------
 builtin_ntp_servers | 127.25.109.148:40813 | all 1 server(s) checked
 time_source         | builtin              | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address        | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
 0aea20ce2a1144519ed144383c016d78 | 127.25.109.132:34649 | HEALTHY | <none>   |       1        |       0
 1864f9ea44c54742b090c594a5be3669 | 127.25.109.131:42119 | HEALTHY | <none>   |       3        |       0
 35f356f20e274db7bf020f78f1bc9ae7 | 127.25.109.130:41639 | HEALTHY | <none>   |       5        |       0
 676092c80dd74040bb49a06db1ac209c | 127.25.109.134:38633 | HEALTHY | <none>   |       1        |       0
 a506969c1a5346c09c19a01f6cbc1a8b | 127.25.109.133:46379 | HEALTHY | <none>   |       4        |       0
 b191086bf5f2432b8d240510817ae5fe | 127.25.109.129:42243 | HEALTHY | <none>   |       2        |       0
 facf1046c8054d2d841cf875c63e29e2 | 127.25.109.135:37253 | HEALTHY | <none>   |       0        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       7

Unusual flags for Tablet Server:
                   Flag                   |                                                                              Value                                                                               |         Tags          |      Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_server_key_size                     | 768                                                                                                                                                              | experimental          | all 7 server(s) checked
 local_ip_for_outbound_sockets            | 127.25.109.129                                                                                                                                                   | experimental          | 127.25.109.129:42243
 local_ip_for_outbound_sockets            | 127.25.109.130                                                                                                                                                   | experimental          | 127.25.109.130:41639
 local_ip_for_outbound_sockets            | 127.25.109.131                                                                                                                                                   | experimental          | 127.25.109.131:42119
 local_ip_for_outbound_sockets            | 127.25.109.132                                                                                                                                                   | experimental          | 127.25.109.132:34649
 local_ip_for_outbound_sockets            | 127.25.109.133                                                                                                                                                   | experimental          | 127.25.109.133:46379
 local_ip_for_outbound_sockets            | 127.25.109.134                                                                                                                                                   | experimental          | 127.25.109.134:38633
 local_ip_for_outbound_sockets            | 127.25.109.135                                                                                                                                                   | experimental          | 127.25.109.135:37253
 never_fsync                              | true                                                                                                                                                             | unsafe,advanced       | all 7 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                | unsafe,hidden         | all 7 server(s) checked
 raft_prepare_replacement_before_eviction | false                                                                                                                                                            | experimental,advanced | all 7 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                             | unsafe                | all 7 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                               | hidden                | all 7 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden                | 127.25.109.129:42243
 server_dump_info_path                    | /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden                | 127.25.109.130:41639
 server_dump_info_path                    | /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden                | 127.25.109.131:42119
 server_dump_info_path                    | /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden                | 127.25.109.132:34649
 server_dump_info_path                    | /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden                | 127.25.109.133:46379
 server_dump_info_path                    | /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden                | 127.25.109.134:38633
 server_dump_info_path                    | /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden                | 127.25.109.135:37253

Flags of checked categories for Tablet Server:
        Flag         |        Value         |      Tablet Server
---------------------+----------------------+-------------------------
 builtin_ntp_servers | 127.25.109.148:40813 | all 7 server(s) checked
 time_source         | builtin              | all 7 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.18.0-SNAPSHOT | all 8 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
          Name          | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
 TestTable              | 2  | HEALTHY | 1             | 1       | 0          | 0                | 0
 rebalance_test_table_0 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_1 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_2 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_3 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_4 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 4
 First Quartile | 4
 Median         | 5
 Third Quartile | 5
 Maximum        | 5

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 7
 Tables         | 6
 Tablets        | 16
 Replicas       | 32

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20240502 01:54:57.368750 26038 log_verifier.cc:126] Checking tablet 02b02a474b5e4f6ab173ced2dc21c03e
I20240502 01:54:57.786072 26038 log_verifier.cc:177] Verified matching terms for 1164 ops in tablet 02b02a474b5e4f6ab173ced2dc21c03e
I20240502 01:54:57.787529 26038 log_verifier.cc:126] Checking tablet 10aa745ed48d46a3b0558d35dee6a807
I20240502 01:54:58.153765 26038 log_verifier.cc:177] Verified matching terms for 1144 ops in tablet 10aa745ed48d46a3b0558d35dee6a807
I20240502 01:54:58.155351 26038 log_verifier.cc:126] Checking tablet 1313391ae5b2440d93445deda2642a14
I20240502 01:54:58.517400 26038 log_verifier.cc:177] Verified matching terms for 1141 ops in tablet 1313391ae5b2440d93445deda2642a14
I20240502 01:54:58.518671 26038 log_verifier.cc:126] Checking tablet 238e56f8d83748e8b2cf1100e92db4c6
I20240502 01:54:58.881569 26038 log_verifier.cc:177] Verified matching terms for 1109 ops in tablet 238e56f8d83748e8b2cf1100e92db4c6
I20240502 01:54:58.882859 26038 log_verifier.cc:126] Checking tablet 4a00f35083c9404ea2ea877826f2fe77
I20240502 01:54:59.257547 26038 log_verifier.cc:177] Verified matching terms for 1178 ops in tablet 4a00f35083c9404ea2ea877826f2fe77
I20240502 01:54:59.258903 26038 log_verifier.cc:126] Checking tablet 5575decad4be448e835db06b570c69a8
I20240502 01:54:59.665825 26038 log_verifier.cc:177] Verified matching terms for 1219 ops in tablet 5575decad4be448e835db06b570c69a8
I20240502 01:54:59.667178 26038 log_verifier.cc:126] Checking tablet 559610821917470fbcf5641d36acf409
I20240502 01:55:00.004679 26038 log_verifier.cc:177] Verified matching terms for 1054 ops in tablet 559610821917470fbcf5641d36acf409
I20240502 01:55:00.005815 26038 log_verifier.cc:126] Checking tablet 65576af2c4bb4f25b4d14963cbe65ca7
I20240502 01:55:00.383662 26038 log_verifier.cc:177] Verified matching terms for 1178 ops in tablet 65576af2c4bb4f25b4d14963cbe65ca7
I20240502 01:55:00.385020 26038 log_verifier.cc:126] Checking tablet 6b56adc64eba4898b80687a2679b70c8
I20240502 01:55:00.792579 26038 log_verifier.cc:177] Verified matching terms for 1180 ops in tablet 6b56adc64eba4898b80687a2679b70c8
I20240502 01:55:00.793798 26038 log_verifier.cc:126] Checking tablet 704700e8872b4675b41d6c172731a5e0
I20240502 01:55:01.173897 26038 log_verifier.cc:177] Verified matching terms for 1178 ops in tablet 704700e8872b4675b41d6c172731a5e0
I20240502 01:55:01.175236 26038 log_verifier.cc:126] Checking tablet 82e5853a509c43df917f257186844ddc
I20240502 01:55:01.566350 26038 log_verifier.cc:177] Verified matching terms for 1150 ops in tablet 82e5853a509c43df917f257186844ddc
I20240502 01:55:01.567829 26038 log_verifier.cc:126] Checking tablet 86f2e43af7fb4e02bb853f0748d62ddb
I20240502 01:55:01.943565 26038 log_verifier.cc:177] Verified matching terms for 1110 ops in tablet 86f2e43af7fb4e02bb853f0748d62ddb
I20240502 01:55:01.945058 26038 log_verifier.cc:126] Checking tablet a865d3891d4e4430a0a0aec06a074cea
I20240502 01:55:02.005599 26038 log_verifier.cc:177] Verified matching terms for 2 ops in tablet a865d3891d4e4430a0a0aec06a074cea
I20240502 01:55:02.005687 26038 log_verifier.cc:126] Checking tablet e714cba54201433caa218073097c1e6a
I20240502 01:55:02.369341 26038 log_verifier.cc:177] Verified matching terms for 1186 ops in tablet e714cba54201433caa218073097c1e6a
I20240502 01:55:02.370718 26038 log_verifier.cc:126] Checking tablet f2a3a225d7f343d9b0ac75e41f0d2a11
I20240502 01:55:02.744191 26038 log_verifier.cc:177] Verified matching terms for 1094 ops in tablet f2a3a225d7f343d9b0ac75e41f0d2a11
I20240502 01:55:02.745427 26038 log_verifier.cc:126] Checking tablet fe2daac9229e44c9bee5f7fbaf886728
I20240502 01:55:03.129637 26038 log_verifier.cc:177] Verified matching terms for 1111 ops in tablet fe2daac9229e44c9bee5f7fbaf886728
I20240502 01:55:03.139344 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.136:0
--local_ip_for_outbound_sockets=127.25.109.136
--webserver_interface=127.25.109.136
--webserver_port=0
--tserver_master_addrs=127.25.109.190:41417
--builtin_ntp_servers=127.25.109.148:40813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 01:55:03.522783  4667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:03.523075  4667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:03.523212  4667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:03.531245  4667 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:55:03.531392  4667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:03.531566  4667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.136
I20240502 01:55:03.541600  4667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:40813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.25.109.136
--webserver_port=0
--tserver_master_addrs=127.25.109.190:41417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.136
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:03.542985  4667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:03.544823  4667 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:03.558216  4672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:03.558708  4673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:03.559242  4675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:03.559521  4667 server_base.cc:1034] running on GCE node
I20240502 01:55:03.560297  4667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:03.561448  4667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:03.562683  4667 hybrid_clock.cc:648] HybridClock initialized: now 1714614903562626 us; error 48 us; skew 500 ppm
I20240502 01:55:03.563076  4667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:03.565558  4667 webserver.cc:460] Webserver started at http://127.25.109.136:40299/ using document root <none> and password file <none>
I20240502 01:55:03.566360  4667 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:03.566483  4667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:03.566835  4667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:55:03.569725  4667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "3f175b3ff87e41dba66ecb07ceb5f896"
format_stamp: "Formatted at 2024-05-02 01:55:03 on dist-test-slave-kj4b"
I20240502 01:55:03.570571  4667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "3f175b3ff87e41dba66ecb07ceb5f896"
format_stamp: "Formatted at 2024-05-02 01:55:03 on dist-test-slave-kj4b"
I20240502 01:55:03.577922  4667 fs_manager.cc:691] Time spent creating directory manager: real 0.007s	user 0.001s	sys 0.007s
I20240502 01:55:03.582479  4681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:03.584252  4667 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.001s
I20240502 01:55:03.584429  4667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "3f175b3ff87e41dba66ecb07ceb5f896"
format_stamp: "Formatted at 2024-05-02 01:55:03 on dist-test-slave-kj4b"
I20240502 01:55:03.584654  4667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:03.602034  4667 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:55:03.602995  4667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:03.603317  4667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:03.604421  4667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:03.605944  4667 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:03.606038  4667 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:03.606150  4667 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:03.606222  4667 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:03.647073  4667 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.136:45685
I20240502 01:55:03.647145  4795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.136:45685 every 8 connection(s)
I20240502 01:55:03.648500  4667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 01:55:03.652756 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 4667
I20240502 01:55:03.652981 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 01:55:03.673971  4796 heartbeater.cc:347] Connected to a master server at 127.25.109.190:41417
I20240502 01:55:03.674309  4796 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:03.675139  4796 heartbeater.cc:511] Master 127.25.109.190:41417 requested a full tablet report, sending...
I20240502 01:55:03.676789 31502 ts_manager.cc:194] Registered new tserver with Master: 3f175b3ff87e41dba66ecb07ceb5f896 (127.25.109.136:45685)
I20240502 01:55:03.677999 31502 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.136:59017
I20240502 01:55:04.191570 31896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:55:04.200193   497 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:04.209532  4729 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:04.220050 31764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:55:04.228371   766 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:04.300943 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:04.310639   630 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:04.355471   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:04.554402 31896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:55:04.555038   497 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:04.557337 31764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:55:04.569244   766 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:04.570874   630 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:04.573350  4729 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:04.595757   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:04.597303 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:04.681290  4796 heartbeater.cc:503] Master 127.25.109.190:41417 was elected leader, sending a full tablet report...
I20240502 01:55:04.736719   649 consensus_queue.cc:237] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1219, Committed index: 1219, Last appended: 2.1219, Last appended by leader: 518, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1220 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } }
I20240502 01:55:04.739189 31647 raft_consensus.cc:1261] T 5575decad4be448e835db06b570c69a8 P b191086bf5f2432b8d240510817ae5fe [term 2 FOLLOWER]: Refusing update from remote peer a506969c1a5346c09c19a01f6cbc1a8b: Log matching property violated. Preceding OpId in replica: term: 2 index: 1219. Preceding OpId from leader: term: 2 index: 1220. (index mismatch)
I20240502 01:55:04.739845  4903 consensus_queue.cc:1035] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1220, Last known committed idx: 1219, Time since last communication: 0.000s
I20240502 01:55:04.742312  4504 raft_consensus.cc:2932] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b [term 2 LEADER]: Committing config change with OpId 2.1220: config changed from index 814 to 1220, VOTER 3f175b3ff87e41dba66ecb07ceb5f896 (127.25.109.136) added. New config: { opid_index: 1220 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } } }
I20240502 01:55:04.743115 31647 raft_consensus.cc:2932] T 5575decad4be448e835db06b570c69a8 P b191086bf5f2432b8d240510817ae5fe [term 2 FOLLOWER]: Committing config change with OpId 2.1220: config changed from index 814 to 1220, VOTER 3f175b3ff87e41dba66ecb07ceb5f896 (127.25.109.136) added. New config: { opid_index: 1220 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } } }
W20240502 01:55:04.747185   583 consensus_peers.cc:485] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b -> Peer 3f175b3ff87e41dba66ecb07ceb5f896 (127.25.109.136:45685): Couldn't send request to peer 3f175b3ff87e41dba66ecb07ceb5f896. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5575decad4be448e835db06b570c69a8. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:04.752894 31501 catalog_manager.cc:5484] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b reported cstate change: config changed from index 814 to 1220, VOTER 3f175b3ff87e41dba66ecb07ceb5f896 (127.25.109.136) added. New cstate: current_term: 2 leader_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" committed_config { opid_index: 1220 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } } }
I20240502 01:55:04.796634   784 consensus_queue.cc:237] T 4a00f35083c9404ea2ea877826f2fe77 P 676092c80dd74040bb49a06db1ac209c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1178, Committed index: 1178, Last appended: 2.1178, Last appended by leader: 404, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1179 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } }
I20240502 01:55:04.799489 31647 raft_consensus.cc:1261] T 4a00f35083c9404ea2ea877826f2fe77 P b191086bf5f2432b8d240510817ae5fe [term 2 FOLLOWER]: Refusing update from remote peer 676092c80dd74040bb49a06db1ac209c: Log matching property violated. Preceding OpId in replica: term: 2 index: 1178. Preceding OpId from leader: term: 2 index: 1179. (index mismatch)
I20240502 01:55:04.800104  4927 consensus_queue.cc:1035] T 4a00f35083c9404ea2ea877826f2fe77 P 676092c80dd74040bb49a06db1ac209c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1179, Last known committed idx: 1178, Time since last communication: 0.000s
I20240502 01:55:04.802675  4802 raft_consensus.cc:2932] T 4a00f35083c9404ea2ea877826f2fe77 P 676092c80dd74040bb49a06db1ac209c [term 2 LEADER]: Committing config change with OpId 2.1179: config changed from index 444 to 1179, VOTER 3f175b3ff87e41dba66ecb07ceb5f896 (127.25.109.136) added. New config: { opid_index: 1179 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } } }
I20240502 01:55:04.803208 31647 raft_consensus.cc:2932] T 4a00f35083c9404ea2ea877826f2fe77 P b191086bf5f2432b8d240510817ae5fe [term 2 FOLLOWER]: Committing config change with OpId 2.1179: config changed from index 444 to 1179, VOTER 3f175b3ff87e41dba66ecb07ceb5f896 (127.25.109.136) added. New config: { opid_index: 1179 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } } }
W20240502 01:55:04.806043   720 consensus_peers.cc:485] T 4a00f35083c9404ea2ea877826f2fe77 P 676092c80dd74040bb49a06db1ac209c -> Peer 3f175b3ff87e41dba66ecb07ceb5f896 (127.25.109.136:45685): Couldn't send request to peer 3f175b3ff87e41dba66ecb07ceb5f896. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4a00f35083c9404ea2ea877826f2fe77. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:04.807813 31499 catalog_manager.cc:5484] T 4a00f35083c9404ea2ea877826f2fe77 P b191086bf5f2432b8d240510817ae5fe reported cstate change: config changed from index 444 to 1179, VOTER 3f175b3ff87e41dba66ecb07ceb5f896 (127.25.109.136) added. New cstate: current_term: 2 leader_uuid: "676092c80dd74040bb49a06db1ac209c" committed_config { opid_index: 1179 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } } }
I20240502 01:55:04.838164 31783 consensus_queue.cc:237] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1109, Committed index: 1109, Last appended: 1.1109, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1110 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } }
I20240502 01:55:04.840766   513 raft_consensus.cc:1261] T 238e56f8d83748e8b2cf1100e92db4c6 P 0aea20ce2a1144519ed144383c016d78 [term 1 FOLLOWER]: Refusing update from remote peer 35f356f20e274db7bf020f78f1bc9ae7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1109. Preceding OpId from leader: term: 1 index: 1110. (index mismatch)
I20240502 01:55:04.841337  3918 consensus_queue.cc:1035] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1110, Last known committed idx: 1109, Time since last communication: 0.000s
I20240502 01:55:04.843978  4949 raft_consensus.cc:2932] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 LEADER]: Committing config change with OpId 1.1110: config changed from index 589 to 1110, VOTER 3f175b3ff87e41dba66ecb07ceb5f896 (127.25.109.136) added. New config: { opid_index: 1110 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } } }
I20240502 01:55:04.844558   513 raft_consensus.cc:2932] T 238e56f8d83748e8b2cf1100e92db4c6 P 0aea20ce2a1144519ed144383c016d78 [term 1 FOLLOWER]: Committing config change with OpId 1.1110: config changed from index 589 to 1110, VOTER 3f175b3ff87e41dba66ecb07ceb5f896 (127.25.109.136) added. New config: { opid_index: 1110 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } } }
W20240502 01:55:04.846741 31718 consensus_peers.cc:485] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7 -> Peer 3f175b3ff87e41dba66ecb07ceb5f896 (127.25.109.136:45685): Couldn't send request to peer 3f175b3ff87e41dba66ecb07ceb5f896. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 238e56f8d83748e8b2cf1100e92db4c6. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:04.849642 31501 catalog_manager.cc:5484] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7 reported cstate change: config changed from index 589 to 1110, VOTER 3f175b3ff87e41dba66ecb07ceb5f896 (127.25.109.136) added. New cstate: current_term: 1 leader_uuid: "35f356f20e274db7bf020f78f1bc9ae7" committed_config { opid_index: 1110 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } } }
I20240502 01:55:04.882375   649 consensus_queue.cc:237] T 65576af2c4bb4f25b4d14963cbe65ca7 P a506969c1a5346c09c19a01f6cbc1a8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1178, Committed index: 1178, Last appended: 2.1178, Last appended by leader: 994, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1179 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } }
I20240502 01:55:04.884806 31913 raft_consensus.cc:1261] T 65576af2c4bb4f25b4d14963cbe65ca7 P 1864f9ea44c54742b090c594a5be3669 [term 2 FOLLOWER]: Refusing update from remote peer a506969c1a5346c09c19a01f6cbc1a8b: Log matching property violated. Preceding OpId in replica: term: 2 index: 1178. Preceding OpId from leader: term: 2 index: 1179. (index mismatch)
W20240502 01:55:04.884972   583 consensus_peers.cc:485] T 65576af2c4bb4f25b4d14963cbe65ca7 P a506969c1a5346c09c19a01f6cbc1a8b -> Peer 3f175b3ff87e41dba66ecb07ceb5f896 (127.25.109.136:45685): Couldn't send request to peer 3f175b3ff87e41dba66ecb07ceb5f896. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 65576af2c4bb4f25b4d14963cbe65ca7. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:04.885414  4504 consensus_queue.cc:1035] T 65576af2c4bb4f25b4d14963cbe65ca7 P a506969c1a5346c09c19a01f6cbc1a8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1179, Last known committed idx: 1178, Time since last communication: 0.000s
I20240502 01:55:04.887897  4903 raft_consensus.cc:2932] T 65576af2c4bb4f25b4d14963cbe65ca7 P a506969c1a5346c09c19a01f6cbc1a8b [term 2 LEADER]: Committing config change with OpId 2.1179: config changed from index 1044 to 1179, VOTER 3f175b3ff87e41dba66ecb07ceb5f896 (127.25.109.136) added. New config: { opid_index: 1179 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } } }
I20240502 01:55:04.888520 31913 raft_consensus.cc:2932] T 65576af2c4bb4f25b4d14963cbe65ca7 P 1864f9ea44c54742b090c594a5be3669 [term 2 FOLLOWER]: Committing config change with OpId 2.1179: config changed from index 1044 to 1179, VOTER 3f175b3ff87e41dba66ecb07ceb5f896 (127.25.109.136) added. New config: { opid_index: 1179 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } } }
I20240502 01:55:04.892979 31499 catalog_manager.cc:5484] T 65576af2c4bb4f25b4d14963cbe65ca7 P 1864f9ea44c54742b090c594a5be3669 reported cstate change: config changed from index 1044 to 1179, VOTER 3f175b3ff87e41dba66ecb07ceb5f896 (127.25.109.136) added. New cstate: current_term: 2 leader_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" committed_config { opid_index: 1179 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } } }
I20240502 01:55:05.000543   497 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:05.001153 31896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:55:05.013758 31764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:55:05.025238  4729 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:05.040012   630 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:05.041194 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:05.045332   766 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:05.074295   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:05.183279  5029 ts_tablet_manager.cc:902] T 5575decad4be448e835db06b570c69a8 P 3f175b3ff87e41dba66ecb07ceb5f896: Initiating tablet copy from peer a506969c1a5346c09c19a01f6cbc1a8b (127.25.109.133:46379)
I20240502 01:55:05.186228  5029 tablet_copy_client.cc:308] T 5575decad4be448e835db06b570c69a8 P 3f175b3ff87e41dba66ecb07ceb5f896: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.133:46379
I20240502 01:55:05.193872   672 tablet_copy_service.cc:140] P a506969c1a5346c09c19a01f6cbc1a8b: Received BeginTabletCopySession request for tablet 5575decad4be448e835db06b570c69a8 from peer 3f175b3ff87e41dba66ecb07ceb5f896 ({username='slave'} at 127.25.109.136:54491)
I20240502 01:55:05.194084   672 tablet_copy_service.cc:161] P a506969c1a5346c09c19a01f6cbc1a8b: Beginning new tablet copy session on tablet 5575decad4be448e835db06b570c69a8 from peer 3f175b3ff87e41dba66ecb07ceb5f896 at {username='slave'} at 127.25.109.136:54491: session id = 3f175b3ff87e41dba66ecb07ceb5f896-5575decad4be448e835db06b570c69a8
I20240502 01:55:05.197121   672 tablet_copy_source_session.cc:205] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:05.202440  5029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5575decad4be448e835db06b570c69a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:05.220026  5029 tablet_copy_client.cc:783] T 5575decad4be448e835db06b570c69a8 P 3f175b3ff87e41dba66ecb07ceb5f896: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:05.220418  5029 tablet_copy_client.cc:647] T 5575decad4be448e835db06b570c69a8 P 3f175b3ff87e41dba66ecb07ceb5f896: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:05.226634  5029 tablet_copy_client.cc:523] T 5575decad4be448e835db06b570c69a8 P 3f175b3ff87e41dba66ecb07ceb5f896: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:05.230500  5029 tablet_bootstrap.cc:492] T 5575decad4be448e835db06b570c69a8 P 3f175b3ff87e41dba66ecb07ceb5f896: Bootstrap starting.
I20240502 01:55:05.255450   497 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:05.269047 31764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:55:05.297000   766 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:05.297634 31896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:55:05.301923  5048 ts_tablet_manager.cc:902] T 4a00f35083c9404ea2ea877826f2fe77 P 3f175b3ff87e41dba66ecb07ceb5f896: Initiating tablet copy from peer 676092c80dd74040bb49a06db1ac209c (127.25.109.134:38633)
I20240502 01:55:05.325229  4729 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:05.329016   630 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:05.353493 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:05.360687  5052 ts_tablet_manager.cc:902] T 65576af2c4bb4f25b4d14963cbe65ca7 P 3f175b3ff87e41dba66ecb07ceb5f896: Initiating tablet copy from peer a506969c1a5346c09c19a01f6cbc1a8b (127.25.109.133:46379)
I20240502 01:55:05.374321   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:05.439378  5056 ts_tablet_manager.cc:902] T 238e56f8d83748e8b2cf1100e92db4c6 P 3f175b3ff87e41dba66ecb07ceb5f896: Initiating tablet copy from peer 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130:41639)
I20240502 01:55:05.441362  5052 tablet_copy_client.cc:308] T 65576af2c4bb4f25b4d14963cbe65ca7 P 3f175b3ff87e41dba66ecb07ceb5f896: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.133:46379
I20240502 01:55:05.442482   672 tablet_copy_service.cc:140] P a506969c1a5346c09c19a01f6cbc1a8b: Received BeginTabletCopySession request for tablet 65576af2c4bb4f25b4d14963cbe65ca7 from peer 3f175b3ff87e41dba66ecb07ceb5f896 ({username='slave'} at 127.25.109.136:54491)
I20240502 01:55:05.442656   672 tablet_copy_service.cc:161] P a506969c1a5346c09c19a01f6cbc1a8b: Beginning new tablet copy session on tablet 65576af2c4bb4f25b4d14963cbe65ca7 from peer 3f175b3ff87e41dba66ecb07ceb5f896 at {username='slave'} at 127.25.109.136:54491: session id = 3f175b3ff87e41dba66ecb07ceb5f896-65576af2c4bb4f25b4d14963cbe65ca7
I20240502 01:55:05.445686   672 tablet_copy_source_session.cc:205] T 65576af2c4bb4f25b4d14963cbe65ca7 P a506969c1a5346c09c19a01f6cbc1a8b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:05.448104  5048 tablet_copy_client.cc:308] T 4a00f35083c9404ea2ea877826f2fe77 P 3f175b3ff87e41dba66ecb07ceb5f896: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.134:38633
I20240502 01:55:05.452459  5052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65576af2c4bb4f25b4d14963cbe65ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:05.459461  5052 tablet_copy_client.cc:783] T 65576af2c4bb4f25b4d14963cbe65ca7 P 3f175b3ff87e41dba66ecb07ceb5f896: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:05.469373   807 tablet_copy_service.cc:140] P 676092c80dd74040bb49a06db1ac209c: Received BeginTabletCopySession request for tablet 4a00f35083c9404ea2ea877826f2fe77 from peer 3f175b3ff87e41dba66ecb07ceb5f896 ({username='slave'} at 127.25.109.136:59593)
I20240502 01:55:05.469641   807 tablet_copy_service.cc:161] P 676092c80dd74040bb49a06db1ac209c: Beginning new tablet copy session on tablet 4a00f35083c9404ea2ea877826f2fe77 from peer 3f175b3ff87e41dba66ecb07ceb5f896 at {username='slave'} at 127.25.109.136:59593: session id = 3f175b3ff87e41dba66ecb07ceb5f896-4a00f35083c9404ea2ea877826f2fe77
I20240502 01:55:05.470939  5052 tablet_copy_client.cc:647] T 65576af2c4bb4f25b4d14963cbe65ca7 P 3f175b3ff87e41dba66ecb07ceb5f896: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:05.473285   807 tablet_copy_source_session.cc:205] T 4a00f35083c9404ea2ea877826f2fe77 P 676092c80dd74040bb49a06db1ac209c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:05.477022  5048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a00f35083c9404ea2ea877826f2fe77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:05.479794  5052 tablet_copy_client.cc:523] T 65576af2c4bb4f25b4d14963cbe65ca7 P 3f175b3ff87e41dba66ecb07ceb5f896: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:05.484200  5052 tablet_bootstrap.cc:492] T 65576af2c4bb4f25b4d14963cbe65ca7 P 3f175b3ff87e41dba66ecb07ceb5f896: Bootstrap starting.
I20240502 01:55:05.487406  5048 tablet_copy_client.cc:783] T 4a00f35083c9404ea2ea877826f2fe77 P 3f175b3ff87e41dba66ecb07ceb5f896: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:05.487800  5048 tablet_copy_client.cc:647] T 4a00f35083c9404ea2ea877826f2fe77 P 3f175b3ff87e41dba66ecb07ceb5f896: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:05.491220  5056 tablet_copy_client.cc:308] T 238e56f8d83748e8b2cf1100e92db4c6 P 3f175b3ff87e41dba66ecb07ceb5f896: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.130:41639
I20240502 01:55:05.502432  5048 tablet_copy_client.cc:523] T 4a00f35083c9404ea2ea877826f2fe77 P 3f175b3ff87e41dba66ecb07ceb5f896: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:05.502872 31805 tablet_copy_service.cc:140] P 35f356f20e274db7bf020f78f1bc9ae7: Received BeginTabletCopySession request for tablet 238e56f8d83748e8b2cf1100e92db4c6 from peer 3f175b3ff87e41dba66ecb07ceb5f896 ({username='slave'} at 127.25.109.136:46941)
I20240502 01:55:05.503083 31805 tablet_copy_service.cc:161] P 35f356f20e274db7bf020f78f1bc9ae7: Beginning new tablet copy session on tablet 238e56f8d83748e8b2cf1100e92db4c6 from peer 3f175b3ff87e41dba66ecb07ceb5f896 at {username='slave'} at 127.25.109.136:46941: session id = 3f175b3ff87e41dba66ecb07ceb5f896-238e56f8d83748e8b2cf1100e92db4c6
I20240502 01:55:05.506163 31805 tablet_copy_source_session.cc:205] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:05.506783  5048 tablet_bootstrap.cc:492] T 4a00f35083c9404ea2ea877826f2fe77 P 3f175b3ff87e41dba66ecb07ceb5f896: Bootstrap starting.
I20240502 01:55:05.508123  5056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 238e56f8d83748e8b2cf1100e92db4c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:05.517771  5056 tablet_copy_client.cc:783] T 238e56f8d83748e8b2cf1100e92db4c6 P 3f175b3ff87e41dba66ecb07ceb5f896: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:05.518738  5056 tablet_copy_client.cc:647] T 238e56f8d83748e8b2cf1100e92db4c6 P 3f175b3ff87e41dba66ecb07ceb5f896: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:05.525461  5056 tablet_copy_client.cc:523] T 238e56f8d83748e8b2cf1100e92db4c6 P 3f175b3ff87e41dba66ecb07ceb5f896: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:05.529896  5056 tablet_bootstrap.cc:492] T 238e56f8d83748e8b2cf1100e92db4c6 P 3f175b3ff87e41dba66ecb07ceb5f896: Bootstrap starting.
I20240502 01:55:05.729526  5029 log.cc:824] T 5575decad4be448e835db06b570c69a8 P 3f175b3ff87e41dba66ecb07ceb5f896: Log is configured to *not* fsync() on all Append() calls
I20240502 01:55:05.812322 31896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:55:05.836830   497 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:05.860949 31764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:55:05.872505   766 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:05.917256  4729 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:05.939575 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:05.959134   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:05.974897   630 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:06.630108  4729 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:06.645593   497 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:06.662632 31764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:55:06.663199   766 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:06.680155   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:06.680670 31896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:55:06.695461 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:06.737221   630 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:07.167457 31896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:55:07.189734   497 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:07.214457   630 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:07.224092 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:07.247222   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:07.250639 31764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:55:07.263195   766 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:07.274988  4729 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:55:07.493371   583 consensus_peers.cc:485] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b -> Peer 3f175b3ff87e41dba66ecb07ceb5f896 (127.25.109.136:45685): Couldn't send request to peer 3f175b3ff87e41dba66ecb07ceb5f896. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:07.649439 31718 consensus_peers.cc:485] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7 -> Peer 3f175b3ff87e41dba66ecb07ceb5f896 (127.25.109.136:45685): Couldn't send request to peer 3f175b3ff87e41dba66ecb07ceb5f896. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:07.708554   720 consensus_peers.cc:485] T 4a00f35083c9404ea2ea877826f2fe77 P 676092c80dd74040bb49a06db1ac209c -> Peer 3f175b3ff87e41dba66ecb07ceb5f896 (127.25.109.136:45685): Couldn't send request to peer 3f175b3ff87e41dba66ecb07ceb5f896. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20240502 01:55:07.787156   583 consensus_peers.cc:485] T 65576af2c4bb4f25b4d14963cbe65ca7 P a506969c1a5346c09c19a01f6cbc1a8b -> Peer 3f175b3ff87e41dba66ecb07ceb5f896 (127.25.109.136:45685): Couldn't send request to peer 3f175b3ff87e41dba66ecb07ceb5f896. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240502 01:55:07.840395   497 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:07.852782 31764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:55:07.922418   766 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:07.933374  4729 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:07.962967 31896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:55:07.983805   630 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:08.026136 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:08.052621   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:08.200685  5056 tablet_bootstrap.cc:492] T 238e56f8d83748e8b2cf1100e92db4c6 P 3f175b3ff87e41dba66ecb07ceb5f896: Bootstrap replayed 1/1 log segments. Stats: ops{read=1110 overwritten=0 applied=1110 ignored=0} inserts{seen=1739 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:08.201481  5056 tablet_bootstrap.cc:492] T 238e56f8d83748e8b2cf1100e92db4c6 P 3f175b3ff87e41dba66ecb07ceb5f896: Bootstrap complete.
I20240502 01:55:08.202268  5056 ts_tablet_manager.cc:1367] T 238e56f8d83748e8b2cf1100e92db4c6 P 3f175b3ff87e41dba66ecb07ceb5f896: Time spent bootstrapping tablet: real 2.672s	user 1.297s	sys 0.105s
I20240502 01:55:08.209702  5056 raft_consensus.cc:348] T 238e56f8d83748e8b2cf1100e92db4c6 P 3f175b3ff87e41dba66ecb07ceb5f896 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 1110 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } }
I20240502 01:55:08.210419  5056 raft_consensus.cc:729] T 238e56f8d83748e8b2cf1100e92db4c6 P 3f175b3ff87e41dba66ecb07ceb5f896 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f175b3ff87e41dba66ecb07ceb5f896, State: Initialized, Role: FOLLOWER
I20240502 01:55:08.211295  5056 consensus_queue.cc:260] T 238e56f8d83748e8b2cf1100e92db4c6 P 3f175b3ff87e41dba66ecb07ceb5f896 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1110, Last appended: 1.1110, Last appended by leader: 1110, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1110 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } }
I20240502 01:55:08.231806  5056 ts_tablet_manager.cc:1398] T 238e56f8d83748e8b2cf1100e92db4c6 P 3f175b3ff87e41dba66ecb07ceb5f896: Time spent starting tablet: real 0.029s	user 0.010s	sys 0.003s
I20240502 01:55:08.233325 31805 tablet_copy_service.cc:342] P 35f356f20e274db7bf020f78f1bc9ae7: Request end of tablet copy session 3f175b3ff87e41dba66ecb07ceb5f896-238e56f8d83748e8b2cf1100e92db4c6 received from {username='slave'} at 127.25.109.136:46941
I20240502 01:55:08.233465 31805 tablet_copy_service.cc:433] P 35f356f20e274db7bf020f78f1bc9ae7: ending tablet copy session 3f175b3ff87e41dba66ecb07ceb5f896-238e56f8d83748e8b2cf1100e92db4c6 on tablet 238e56f8d83748e8b2cf1100e92db4c6 with peer 3f175b3ff87e41dba66ecb07ceb5f896
I20240502 01:55:08.323372  5029 tablet_bootstrap.cc:492] T 5575decad4be448e835db06b570c69a8 P 3f175b3ff87e41dba66ecb07ceb5f896: Bootstrap replayed 1/1 log segments. Stats: ops{read=1220 overwritten=0 applied=1220 ignored=0} inserts{seen=1848 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:08.324110  5029 tablet_bootstrap.cc:492] T 5575decad4be448e835db06b570c69a8 P 3f175b3ff87e41dba66ecb07ceb5f896: Bootstrap complete.
I20240502 01:55:08.324596  5029 ts_tablet_manager.cc:1367] T 5575decad4be448e835db06b570c69a8 P 3f175b3ff87e41dba66ecb07ceb5f896: Time spent bootstrapping tablet: real 3.094s	user 1.435s	sys 0.136s
I20240502 01:55:08.325795  5029 raft_consensus.cc:348] T 5575decad4be448e835db06b570c69a8 P 3f175b3ff87e41dba66ecb07ceb5f896 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 1220 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } }
I20240502 01:55:08.326125  5029 raft_consensus.cc:729] T 5575decad4be448e835db06b570c69a8 P 3f175b3ff87e41dba66ecb07ceb5f896 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f175b3ff87e41dba66ecb07ceb5f896, State: Initialized, Role: FOLLOWER
I20240502 01:55:08.326524  5029 consensus_queue.cc:260] T 5575decad4be448e835db06b570c69a8 P 3f175b3ff87e41dba66ecb07ceb5f896 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1220, Last appended: 2.1220, Last appended by leader: 1220, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1220 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } }
I20240502 01:55:08.327172  5029 ts_tablet_manager.cc:1398] T 5575decad4be448e835db06b570c69a8 P 3f175b3ff87e41dba66ecb07ceb5f896: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:55:08.328292   672 tablet_copy_service.cc:342] P a506969c1a5346c09c19a01f6cbc1a8b: Request end of tablet copy session 3f175b3ff87e41dba66ecb07ceb5f896-5575decad4be448e835db06b570c69a8 received from {username='slave'} at 127.25.109.136:54491
I20240502 01:55:08.328446   672 tablet_copy_service.cc:433] P a506969c1a5346c09c19a01f6cbc1a8b: ending tablet copy session 3f175b3ff87e41dba66ecb07ceb5f896-5575decad4be448e835db06b570c69a8 on tablet 5575decad4be448e835db06b570c69a8 with peer 3f175b3ff87e41dba66ecb07ceb5f896
I20240502 01:55:08.369247  5052 tablet_bootstrap.cc:492] T 65576af2c4bb4f25b4d14963cbe65ca7 P 3f175b3ff87e41dba66ecb07ceb5f896: Bootstrap replayed 1/1 log segments. Stats: ops{read=1179 overwritten=0 applied=1179 ignored=0} inserts{seen=1861 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:08.369904  5052 tablet_bootstrap.cc:492] T 65576af2c4bb4f25b4d14963cbe65ca7 P 3f175b3ff87e41dba66ecb07ceb5f896: Bootstrap complete.
I20240502 01:55:08.370419  5052 ts_tablet_manager.cc:1367] T 65576af2c4bb4f25b4d14963cbe65ca7 P 3f175b3ff87e41dba66ecb07ceb5f896: Time spent bootstrapping tablet: real 2.886s	user 1.366s	sys 0.097s
I20240502 01:55:08.371456  5052 raft_consensus.cc:348] T 65576af2c4bb4f25b4d14963cbe65ca7 P 3f175b3ff87e41dba66ecb07ceb5f896 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 1179 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } }
I20240502 01:55:08.371784  5052 raft_consensus.cc:729] T 65576af2c4bb4f25b4d14963cbe65ca7 P 3f175b3ff87e41dba66ecb07ceb5f896 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f175b3ff87e41dba66ecb07ceb5f896, State: Initialized, Role: FOLLOWER
I20240502 01:55:08.372200  5052 consensus_queue.cc:260] T 65576af2c4bb4f25b4d14963cbe65ca7 P 3f175b3ff87e41dba66ecb07ceb5f896 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1179, Last appended: 2.1179, Last appended by leader: 1179, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1179 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } }
I20240502 01:55:08.373028  5052 ts_tablet_manager.cc:1398] T 65576af2c4bb4f25b4d14963cbe65ca7 P 3f175b3ff87e41dba66ecb07ceb5f896: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:55:08.374050   672 tablet_copy_service.cc:342] P a506969c1a5346c09c19a01f6cbc1a8b: Request end of tablet copy session 3f175b3ff87e41dba66ecb07ceb5f896-65576af2c4bb4f25b4d14963cbe65ca7 received from {username='slave'} at 127.25.109.136:54491
I20240502 01:55:08.374182   672 tablet_copy_service.cc:433] P a506969c1a5346c09c19a01f6cbc1a8b: ending tablet copy session 3f175b3ff87e41dba66ecb07ceb5f896-65576af2c4bb4f25b4d14963cbe65ca7 on tablet 65576af2c4bb4f25b4d14963cbe65ca7 with peer 3f175b3ff87e41dba66ecb07ceb5f896
I20240502 01:55:08.400985  4749 raft_consensus.cc:1203] T 5575decad4be448e835db06b570c69a8 P 3f175b3ff87e41dba66ecb07ceb5f896 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.1219->[2.1220-2.1220]   Dedup: 2.1220->[]
I20240502 01:55:08.481515  5048 tablet_bootstrap.cc:492] T 4a00f35083c9404ea2ea877826f2fe77 P 3f175b3ff87e41dba66ecb07ceb5f896: Bootstrap replayed 1/1 log segments. Stats: ops{read=1179 overwritten=0 applied=1179 ignored=0} inserts{seen=1848 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:08.481948  5048 tablet_bootstrap.cc:492] T 4a00f35083c9404ea2ea877826f2fe77 P 3f175b3ff87e41dba66ecb07ceb5f896: Bootstrap complete.
I20240502 01:55:08.482277  5048 ts_tablet_manager.cc:1367] T 4a00f35083c9404ea2ea877826f2fe77 P 3f175b3ff87e41dba66ecb07ceb5f896: Time spent bootstrapping tablet: real 2.976s	user 1.308s	sys 0.141s
I20240502 01:55:08.482947  5048 raft_consensus.cc:348] T 4a00f35083c9404ea2ea877826f2fe77 P 3f175b3ff87e41dba66ecb07ceb5f896 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 1179 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } }
I20240502 01:55:08.483160  5048 raft_consensus.cc:729] T 4a00f35083c9404ea2ea877826f2fe77 P 3f175b3ff87e41dba66ecb07ceb5f896 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f175b3ff87e41dba66ecb07ceb5f896, State: Initialized, Role: FOLLOWER
I20240502 01:55:08.483428  5048 consensus_queue.cc:260] T 4a00f35083c9404ea2ea877826f2fe77 P 3f175b3ff87e41dba66ecb07ceb5f896 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1179, Last appended: 2.1179, Last appended by leader: 1179, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1179 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } }
I20240502 01:55:08.483992  5048 ts_tablet_manager.cc:1398] T 4a00f35083c9404ea2ea877826f2fe77 P 3f175b3ff87e41dba66ecb07ceb5f896: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240502 01:55:08.484921   807 tablet_copy_service.cc:342] P 676092c80dd74040bb49a06db1ac209c: Request end of tablet copy session 3f175b3ff87e41dba66ecb07ceb5f896-4a00f35083c9404ea2ea877826f2fe77 received from {username='slave'} at 127.25.109.136:59593
I20240502 01:55:08.485100   807 tablet_copy_service.cc:433] P 676092c80dd74040bb49a06db1ac209c: ending tablet copy session 3f175b3ff87e41dba66ecb07ceb5f896-4a00f35083c9404ea2ea877826f2fe77 on tablet 4a00f35083c9404ea2ea877826f2fe77 with peer 3f175b3ff87e41dba66ecb07ceb5f896
I20240502 01:55:08.517195   497 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:08.529194 31896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:55:08.541010 31764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:55:08.591667   630 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:08.595389  4749 raft_consensus.cc:1203] T 238e56f8d83748e8b2cf1100e92db4c6 P 3f175b3ff87e41dba66ecb07ceb5f896 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1109->[1.1110-1.1110]   Dedup: 1.1110->[]
I20240502 01:55:08.595909  4729 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:08.604444 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:08.634776   766 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:08.658645   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:08.781622  4749 raft_consensus.cc:1203] T 65576af2c4bb4f25b4d14963cbe65ca7 P 3f175b3ff87e41dba66ecb07ceb5f896 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.1178->[2.1179-2.1179]   Dedup: 2.1179->[]
I20240502 01:55:08.834184 31764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:55:08.835444   497 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:08.850072   766 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:08.851668 31896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:55:08.852712  4729 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:08.877569 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:08.888469   630 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:08.907729   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:08.925503  4749 raft_consensus.cc:1203] T 4a00f35083c9404ea2ea877826f2fe77 P 3f175b3ff87e41dba66ecb07ceb5f896 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.1178->[2.1179-2.1179]   Dedup: 2.1179->[]
I20240502 01:55:08.988287 31783 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "238e56f8d83748e8b2cf1100e92db4c6"
dest_uuid: "35f356f20e274db7bf020f78f1bc9ae7"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:57634
I20240502 01:55:08.988502 31783 raft_consensus.cc:595] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:55:09.075662   497 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:09.089183 31764 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:55:09.100137  4729 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:09.102468   766 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:09.103415 31896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:55:09.135069 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:09.135929   630 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:09.146070   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:09.188326  5372 raft_consensus.cc:979] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7: : Instructing follower 3f175b3ff87e41dba66ecb07ceb5f896 to start an election
I20240502 01:55:09.188541  5372 raft_consensus.cc:1067] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 LEADER]: Signalling peer 3f175b3ff87e41dba66ecb07ceb5f896 to start an election
I20240502 01:55:09.189553  4749 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "238e56f8d83748e8b2cf1100e92db4c6"
dest_uuid: "3f175b3ff87e41dba66ecb07ceb5f896"
 from {username='slave'} at 127.25.109.130:51385
I20240502 01:55:09.189910  4749 raft_consensus.cc:482] T 238e56f8d83748e8b2cf1100e92db4c6 P 3f175b3ff87e41dba66ecb07ceb5f896 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:55:09.190109  4749 raft_consensus.cc:3037] T 238e56f8d83748e8b2cf1100e92db4c6 P 3f175b3ff87e41dba66ecb07ceb5f896 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:09.193513  4749 raft_consensus.cc:504] T 238e56f8d83748e8b2cf1100e92db4c6 P 3f175b3ff87e41dba66ecb07ceb5f896 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 1110 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } }
I20240502 01:55:09.196991 31783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "238e56f8d83748e8b2cf1100e92db4c6" candidate_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" candidate_term: 2 candidate_status { last_received { term: 1 index: 1110 } } ignore_live_leader: true dest_uuid: "35f356f20e274db7bf020f78f1bc9ae7"
I20240502 01:55:09.197203 31783 raft_consensus.cc:3032] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:55:09.197289 31783 raft_consensus.cc:729] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 35f356f20e274db7bf020f78f1bc9ae7, State: Running, Role: LEADER
I20240502 01:55:09.197561 31783 consensus_queue.cc:260] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1110, Committed index: 1110, Last appended: 1.1110, Last appended by leader: 1110, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1110 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } }
I20240502 01:55:09.197893 31783 raft_consensus.cc:3037] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:09.202356  4749 leader_election.cc:288] T 238e56f8d83748e8b2cf1100e92db4c6 P 3f175b3ff87e41dba66ecb07ceb5f896 [CANDIDATE]: Term 2 election: Requested vote from peers 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130:41639), 0aea20ce2a1144519ed144383c016d78 (127.25.109.132:34649)
I20240502 01:55:09.215415 31783 raft_consensus.cc:2454] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f175b3ff87e41dba66ecb07ceb5f896 in term 2.
I20240502 01:55:09.220299  4683 leader_election.cc:302] T 238e56f8d83748e8b2cf1100e92db4c6 P 3f175b3ff87e41dba66ecb07ceb5f896 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35f356f20e274db7bf020f78f1bc9ae7, 3f175b3ff87e41dba66ecb07ceb5f896; no voters: 
I20240502 01:55:09.220803  5240 raft_consensus.cc:2781] T 238e56f8d83748e8b2cf1100e92db4c6 P 3f175b3ff87e41dba66ecb07ceb5f896 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:55:09.232944  5240 raft_consensus.cc:686] T 238e56f8d83748e8b2cf1100e92db4c6 P 3f175b3ff87e41dba66ecb07ceb5f896 [term 2 LEADER]: Becoming Leader. State: Replica: 3f175b3ff87e41dba66ecb07ceb5f896, State: Running, Role: LEADER
I20240502 01:55:09.233698  5240 consensus_queue.cc:237] T 238e56f8d83748e8b2cf1100e92db4c6 P 3f175b3ff87e41dba66ecb07ceb5f896 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1110, Committed index: 1110, Last appended: 1.1110, Last appended by leader: 1110, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1110 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } }
I20240502 01:55:09.241178   513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "238e56f8d83748e8b2cf1100e92db4c6" candidate_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" candidate_term: 2 candidate_status { last_received { term: 1 index: 1110 } } ignore_live_leader: true dest_uuid: "0aea20ce2a1144519ed144383c016d78"
I20240502 01:55:09.241412   513 raft_consensus.cc:3037] T 238e56f8d83748e8b2cf1100e92db4c6 P 0aea20ce2a1144519ed144383c016d78 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:09.244208   513 raft_consensus.cc:2454] T 238e56f8d83748e8b2cf1100e92db4c6 P 0aea20ce2a1144519ed144383c016d78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f175b3ff87e41dba66ecb07ceb5f896 in term 2.
I20240502 01:55:09.244515 31505 catalog_manager.cc:5484] T 238e56f8d83748e8b2cf1100e92db4c6 P 3f175b3ff87e41dba66ecb07ceb5f896 reported cstate change: term changed from 1 to 2, leader changed from 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130) to 3f175b3ff87e41dba66ecb07ceb5f896 (127.25.109.136). New cstate: current_term: 2 leader_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" committed_config { opid_index: 1110 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } } }
I20240502 01:55:09.385411   497 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:09.394488 31764 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:09.397706 31896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:55:09.423144   766 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:09.430853  4729 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:09.440604 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:09.463702   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:09.474678   630 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:09.541334   649 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5575decad4be448e835db06b570c69a8"
dest_uuid: "a506969c1a5346c09c19a01f6cbc1a8b"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:41458
I20240502 01:55:09.541577   649 raft_consensus.cc:595] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b [term 2 LEADER]: Received request to transfer leadership
I20240502 01:55:09.646492   497 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:09.649049 31896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:55:09.657644  4729 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:09.676146 31764 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:09.680341   630 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:09.682938 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:09.724228   766 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:09.749549   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:09.823904 31783 raft_consensus.cc:1261] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7 [term 2 FOLLOWER]: Refusing update from remote peer 3f175b3ff87e41dba66ecb07ceb5f896: Log matching property violated. Preceding OpId in replica: term: 1 index: 1110. Preceding OpId from leader: term: 2 index: 1111. (index mismatch)
I20240502 01:55:09.824692  5472 consensus_queue.cc:1035] T 238e56f8d83748e8b2cf1100e92db4c6 P 3f175b3ff87e41dba66ecb07ceb5f896 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1111, Last known committed idx: 1110, Time since last communication: 0.000s
I20240502 01:55:09.830519   513 raft_consensus.cc:1261] T 238e56f8d83748e8b2cf1100e92db4c6 P 0aea20ce2a1144519ed144383c016d78 [term 2 FOLLOWER]: Refusing update from remote peer 3f175b3ff87e41dba66ecb07ceb5f896: Log matching property violated. Preceding OpId in replica: term: 1 index: 1110. Preceding OpId from leader: term: 2 index: 1111. (index mismatch)
I20240502 01:55:09.831369  5472 consensus_queue.cc:1035] T 238e56f8d83748e8b2cf1100e92db4c6 P 3f175b3ff87e41dba66ecb07ceb5f896 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1111, Last known committed idx: 1110, Time since last communication: 0.000s
I20240502 01:55:09.899787  5495 raft_consensus.cc:979] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b: : Instructing follower 3f175b3ff87e41dba66ecb07ceb5f896 to start an election
I20240502 01:55:09.900074  5305 raft_consensus.cc:1067] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b [term 2 LEADER]: Signalling peer 3f175b3ff87e41dba66ecb07ceb5f896 to start an election
I20240502 01:55:09.900969  4749 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5575decad4be448e835db06b570c69a8"
dest_uuid: "3f175b3ff87e41dba66ecb07ceb5f896"
 from {username='slave'} at 127.25.109.133:38711
I20240502 01:55:09.901187  4749 raft_consensus.cc:482] T 5575decad4be448e835db06b570c69a8 P 3f175b3ff87e41dba66ecb07ceb5f896 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:55:09.901293  4749 raft_consensus.cc:3037] T 5575decad4be448e835db06b570c69a8 P 3f175b3ff87e41dba66ecb07ceb5f896 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:55:09.903901  4749 raft_consensus.cc:504] T 5575decad4be448e835db06b570c69a8 P 3f175b3ff87e41dba66ecb07ceb5f896 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 1220 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } }
I20240502 01:55:09.906420  4749 leader_election.cc:288] T 5575decad4be448e835db06b570c69a8 P 3f175b3ff87e41dba66ecb07ceb5f896 [CANDIDATE]: Term 3 election: Requested vote from peers b191086bf5f2432b8d240510817ae5fe (127.25.109.129:42243), a506969c1a5346c09c19a01f6cbc1a8b (127.25.109.133:46379)
I20240502 01:55:09.906828   649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5575decad4be448e835db06b570c69a8" candidate_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" candidate_term: 3 candidate_status { last_received { term: 2 index: 1220 } } ignore_live_leader: true dest_uuid: "a506969c1a5346c09c19a01f6cbc1a8b"
I20240502 01:55:09.907024   649 raft_consensus.cc:3032] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:55:09.907090   649 raft_consensus.cc:729] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a506969c1a5346c09c19a01f6cbc1a8b, State: Running, Role: LEADER
I20240502 01:55:09.907368   649 consensus_queue.cc:260] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1220, Committed index: 1220, Last appended: 2.1220, Last appended by leader: 1220, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1220 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } }
I20240502 01:55:09.907704   649 raft_consensus.cc:3037] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:55:09.910595   649 raft_consensus.cc:2454] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f175b3ff87e41dba66ecb07ceb5f896 in term 3.
I20240502 01:55:09.911134  4685 leader_election.cc:302] T 5575decad4be448e835db06b570c69a8 P 3f175b3ff87e41dba66ecb07ceb5f896 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f175b3ff87e41dba66ecb07ceb5f896, a506969c1a5346c09c19a01f6cbc1a8b; no voters: 
I20240502 01:55:09.911463  5474 raft_consensus.cc:2781] T 5575decad4be448e835db06b570c69a8 P 3f175b3ff87e41dba66ecb07ceb5f896 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:55:09.911603  5474 raft_consensus.cc:686] T 5575decad4be448e835db06b570c69a8 P 3f175b3ff87e41dba66ecb07ceb5f896 [term 3 LEADER]: Becoming Leader. State: Replica: 3f175b3ff87e41dba66ecb07ceb5f896, State: Running, Role: LEADER
I20240502 01:55:09.911924  5474 consensus_queue.cc:237] T 5575decad4be448e835db06b570c69a8 P 3f175b3ff87e41dba66ecb07ceb5f896 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1220, Committed index: 1220, Last appended: 2.1220, Last appended by leader: 1220, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1220 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } }
I20240502 01:55:09.916852 31500 catalog_manager.cc:5484] T 5575decad4be448e835db06b570c69a8 P 3f175b3ff87e41dba66ecb07ceb5f896 reported cstate change: term changed from 2 to 3, leader changed from a506969c1a5346c09c19a01f6cbc1a8b (127.25.109.133) to 3f175b3ff87e41dba66ecb07ceb5f896 (127.25.109.136). New cstate: current_term: 3 leader_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" committed_config { opid_index: 1220 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } } }
I20240502 01:55:09.922067 31647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5575decad4be448e835db06b570c69a8" candidate_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" candidate_term: 3 candidate_status { last_received { term: 2 index: 1220 } } ignore_live_leader: true dest_uuid: "b191086bf5f2432b8d240510817ae5fe"
I20240502 01:55:09.922276 31647 raft_consensus.cc:3037] T 5575decad4be448e835db06b570c69a8 P b191086bf5f2432b8d240510817ae5fe [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:55:09.925258 31647 raft_consensus.cc:2454] T 5575decad4be448e835db06b570c69a8 P b191086bf5f2432b8d240510817ae5fe [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f175b3ff87e41dba66ecb07ceb5f896 in term 3.
I20240502 01:55:09.928562   497 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:09.941802 31896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:55:09.971159 31764 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:09.972527  4729 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:09.983902   766 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:09.984375   630 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:55:10.007608   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:10.007786 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:10.100507   784 consensus_queue.cc:237] T 4a00f35083c9404ea2ea877826f2fe77 P 676092c80dd74040bb49a06db1ac209c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1179, Committed index: 1179, Last appended: 2.1179, Last appended by leader: 404, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1180 OBSOLETE_local: false peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } }
I20240502 01:55:10.102515  4749 raft_consensus.cc:1261] T 4a00f35083c9404ea2ea877826f2fe77 P 3f175b3ff87e41dba66ecb07ceb5f896 [term 2 FOLLOWER]: Refusing update from remote peer 676092c80dd74040bb49a06db1ac209c: Log matching property violated. Preceding OpId in replica: term: 2 index: 1179. Preceding OpId from leader: term: 2 index: 1180. (index mismatch)
I20240502 01:55:10.103106  5422 consensus_queue.cc:1035] T 4a00f35083c9404ea2ea877826f2fe77 P 676092c80dd74040bb49a06db1ac209c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1180, Last known committed idx: 1179, Time since last communication: 0.000s
I20240502 01:55:10.106302  5529 raft_consensus.cc:2932] T 4a00f35083c9404ea2ea877826f2fe77 P 676092c80dd74040bb49a06db1ac209c [term 2 LEADER]: Committing config change with OpId 2.1180: config changed from index 1179 to 1180, VOTER b191086bf5f2432b8d240510817ae5fe (127.25.109.129) evicted. New config: { opid_index: 1180 OBSOLETE_local: false peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } } }
I20240502 01:55:10.106961  4749 raft_consensus.cc:2932] T 4a00f35083c9404ea2ea877826f2fe77 P 3f175b3ff87e41dba66ecb07ceb5f896 [term 2 FOLLOWER]: Committing config change with OpId 2.1180: config changed from index 1179 to 1180, VOTER b191086bf5f2432b8d240510817ae5fe (127.25.109.129) evicted. New config: { opid_index: 1180 OBSOLETE_local: false peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } } }
I20240502 01:55:10.111442 31500 catalog_manager.cc:5484] T 4a00f35083c9404ea2ea877826f2fe77 P 676092c80dd74040bb49a06db1ac209c reported cstate change: config changed from index 1179 to 1180, VOTER b191086bf5f2432b8d240510817ae5fe (127.25.109.129) evicted. New cstate: current_term: 2 leader_uuid: "676092c80dd74040bb49a06db1ac209c" committed_config { opid_index: 1180 OBSOLETE_local: false peers { permanent_uuid: "676092c80dd74040bb49a06db1ac209c" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 38633 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } } }
I20240502 01:55:10.115748 31628 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a00f35083c9404ea2ea877826f2fe77 with delete_type TABLET_DATA_TOMBSTONED (TS b191086bf5f2432b8d240510817ae5fe not found in new config with opid_index 1180) from {username='slave'} at 127.0.0.1:48324
I20240502 01:55:10.116761  5530 tablet_replica.cc:330] T 4a00f35083c9404ea2ea877826f2fe77 P b191086bf5f2432b8d240510817ae5fe: stopping tablet replica
I20240502 01:55:10.116926  5530 raft_consensus.cc:2229] T 4a00f35083c9404ea2ea877826f2fe77 P b191086bf5f2432b8d240510817ae5fe [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:10.117126  5530 raft_consensus.cc:2258] T 4a00f35083c9404ea2ea877826f2fe77 P b191086bf5f2432b8d240510817ae5fe [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:10.118238  5530 ts_tablet_manager.cc:1875] T 4a00f35083c9404ea2ea877826f2fe77 P b191086bf5f2432b8d240510817ae5fe: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:10.122882  5530 ts_tablet_manager.cc:1888] T 4a00f35083c9404ea2ea877826f2fe77 P b191086bf5f2432b8d240510817ae5fe: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.1179
I20240502 01:55:10.122990  5530 log.cc:1196] T 4a00f35083c9404ea2ea877826f2fe77 P b191086bf5f2432b8d240510817ae5fe: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal/wals/4a00f35083c9404ea2ea877826f2fe77
I20240502 01:55:10.123801 31490 catalog_manager.cc:4830] TS b191086bf5f2432b8d240510817ae5fe (127.25.109.129:42243): tablet 4a00f35083c9404ea2ea877826f2fe77 (table rebalance_test_table_4 [id=9a09fee1a01c4c1c836c0dab06832248]) successfully deleted
I20240502 01:55:10.306151 31647 raft_consensus.cc:1261] T 5575decad4be448e835db06b570c69a8 P b191086bf5f2432b8d240510817ae5fe [term 3 FOLLOWER]: Refusing update from remote peer 3f175b3ff87e41dba66ecb07ceb5f896: Log matching property violated. Preceding OpId in replica: term: 2 index: 1220. Preceding OpId from leader: term: 3 index: 1221. (index mismatch)
I20240502 01:55:10.306766  5474 consensus_queue.cc:1035] T 5575decad4be448e835db06b570c69a8 P 3f175b3ff87e41dba66ecb07ceb5f896 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1221, Last known committed idx: 1220, Time since last communication: 0.000s
I20240502 01:55:10.310798   649 raft_consensus.cc:1261] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b [term 3 FOLLOWER]: Refusing update from remote peer 3f175b3ff87e41dba66ecb07ceb5f896: Log matching property violated. Preceding OpId in replica: term: 2 index: 1220. Preceding OpId from leader: term: 3 index: 1221. (index mismatch)
I20240502 01:55:10.311584  5474 consensus_queue.cc:1035] T 5575decad4be448e835db06b570c69a8 P 3f175b3ff87e41dba66ecb07ceb5f896 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1221, Last known committed idx: 1220, Time since last communication: 0.000s
I20240502 01:55:10.425014   497 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:10.430492 31896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:55:10.436061  4729 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:10.440389   766 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:10.452823   630 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:55:10.455786   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:10.456526 31764 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:10.478740 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:10.574967   649 consensus_queue.cc:237] T 65576af2c4bb4f25b4d14963cbe65ca7 P a506969c1a5346c09c19a01f6cbc1a8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1179, Committed index: 1179, Last appended: 2.1179, Last appended by leader: 994, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1180 OBSOLETE_local: false peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } }
I20240502 01:55:10.577212  4749 raft_consensus.cc:1261] T 65576af2c4bb4f25b4d14963cbe65ca7 P 3f175b3ff87e41dba66ecb07ceb5f896 [term 2 FOLLOWER]: Refusing update from remote peer a506969c1a5346c09c19a01f6cbc1a8b: Log matching property violated. Preceding OpId in replica: term: 2 index: 1179. Preceding OpId from leader: term: 2 index: 1180. (index mismatch)
I20240502 01:55:10.577834  5495 consensus_queue.cc:1035] T 65576af2c4bb4f25b4d14963cbe65ca7 P a506969c1a5346c09c19a01f6cbc1a8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1180, Last known committed idx: 1179, Time since last communication: 0.000s
I20240502 01:55:10.581734  5591 raft_consensus.cc:2932] T 65576af2c4bb4f25b4d14963cbe65ca7 P a506969c1a5346c09c19a01f6cbc1a8b [term 2 LEADER]: Committing config change with OpId 2.1180: config changed from index 1179 to 1180, VOTER 1864f9ea44c54742b090c594a5be3669 (127.25.109.131) evicted. New config: { opid_index: 1180 OBSOLETE_local: false peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } } }
I20240502 01:55:10.582428  4749 raft_consensus.cc:2932] T 65576af2c4bb4f25b4d14963cbe65ca7 P 3f175b3ff87e41dba66ecb07ceb5f896 [term 2 FOLLOWER]: Committing config change with OpId 2.1180: config changed from index 1179 to 1180, VOTER 1864f9ea44c54742b090c594a5be3669 (127.25.109.131) evicted. New config: { opid_index: 1180 OBSOLETE_local: false peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } } }
I20240502 01:55:10.588117 31505 catalog_manager.cc:5484] T 65576af2c4bb4f25b4d14963cbe65ca7 P 3f175b3ff87e41dba66ecb07ceb5f896 reported cstate change: config changed from index 1179 to 1180, VOTER 1864f9ea44c54742b090c594a5be3669 (127.25.109.131) evicted. New cstate: current_term: 2 leader_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" committed_config { opid_index: 1180 OBSOLETE_local: false peers { permanent_uuid: "a506969c1a5346c09c19a01f6cbc1a8b" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 46379 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } } }
I20240502 01:55:10.592154 31896 tablet_service.cc:1514] Processing DeleteTablet for tablet 65576af2c4bb4f25b4d14963cbe65ca7 with delete_type TABLET_DATA_TOMBSTONED (TS 1864f9ea44c54742b090c594a5be3669 not found in new config with opid_index 1180) from {username='slave'} at 127.0.0.1:40670
I20240502 01:55:10.592779  5593 tablet_replica.cc:330] T 65576af2c4bb4f25b4d14963cbe65ca7 P 1864f9ea44c54742b090c594a5be3669: stopping tablet replica
I20240502 01:55:10.592980  5593 raft_consensus.cc:2229] T 65576af2c4bb4f25b4d14963cbe65ca7 P 1864f9ea44c54742b090c594a5be3669 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:10.593220  5593 raft_consensus.cc:2258] T 65576af2c4bb4f25b4d14963cbe65ca7 P 1864f9ea44c54742b090c594a5be3669 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:10.594735  5593 ts_tablet_manager.cc:1875] T 65576af2c4bb4f25b4d14963cbe65ca7 P 1864f9ea44c54742b090c594a5be3669: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:10.600204  5593 ts_tablet_manager.cc:1888] T 65576af2c4bb4f25b4d14963cbe65ca7 P 1864f9ea44c54742b090c594a5be3669: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.1179
I20240502 01:55:10.600353  5593 log.cc:1196] T 65576af2c4bb4f25b4d14963cbe65ca7 P 1864f9ea44c54742b090c594a5be3669: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/wals/65576af2c4bb4f25b4d14963cbe65ca7
I20240502 01:55:10.601606 31489 catalog_manager.cc:4830] TS 1864f9ea44c54742b090c594a5be3669 (127.25.109.131:42119): tablet 65576af2c4bb4f25b4d14963cbe65ca7 (table rebalance_test_table_2 [id=7340ad22b7574a6d81f7eaa977a06414]) successfully deleted
I20240502 01:55:10.736982   497 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:10.747198   630 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:55:10.749050  4729 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:10.751150 31764 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:10.757619 31896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:55:10.780421   766 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:10.793419 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:10.812741   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:10.904438  4749 consensus_queue.cc:237] T 238e56f8d83748e8b2cf1100e92db4c6 P 3f175b3ff87e41dba66ecb07ceb5f896 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1111, Committed index: 1111, Last appended: 2.1111, Last appended by leader: 1110, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1112 OBSOLETE_local: false peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } }
I20240502 01:55:10.906399   513 raft_consensus.cc:1261] T 238e56f8d83748e8b2cf1100e92db4c6 P 0aea20ce2a1144519ed144383c016d78 [term 2 FOLLOWER]: Refusing update from remote peer 3f175b3ff87e41dba66ecb07ceb5f896: Log matching property violated. Preceding OpId in replica: term: 2 index: 1111. Preceding OpId from leader: term: 2 index: 1112. (index mismatch)
I20240502 01:55:10.906911  5474 consensus_queue.cc:1035] T 238e56f8d83748e8b2cf1100e92db4c6 P 3f175b3ff87e41dba66ecb07ceb5f896 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1112, Last known committed idx: 1111, Time since last communication: 0.000s
I20240502 01:55:10.909526  5650 raft_consensus.cc:2932] T 238e56f8d83748e8b2cf1100e92db4c6 P 3f175b3ff87e41dba66ecb07ceb5f896 [term 2 LEADER]: Committing config change with OpId 2.1112: config changed from index 1110 to 1112, VOTER 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130) evicted. New config: { opid_index: 1112 OBSOLETE_local: false peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } } }
I20240502 01:55:10.910079   513 raft_consensus.cc:2932] T 238e56f8d83748e8b2cf1100e92db4c6 P 0aea20ce2a1144519ed144383c016d78 [term 2 FOLLOWER]: Committing config change with OpId 2.1112: config changed from index 1110 to 1112, VOTER 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130) evicted. New config: { opid_index: 1112 OBSOLETE_local: false peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } } }
I20240502 01:55:10.915464 31499 catalog_manager.cc:5484] T 238e56f8d83748e8b2cf1100e92db4c6 P 3f175b3ff87e41dba66ecb07ceb5f896 reported cstate change: config changed from index 1110 to 1112, VOTER 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130) evicted. New cstate: current_term: 2 leader_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" committed_config { opid_index: 1112 OBSOLETE_local: false peers { permanent_uuid: "0aea20ce2a1144519ed144383c016d78" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 34649 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } } }
I20240502 01:55:10.919057 31764 tablet_service.cc:1514] Processing DeleteTablet for tablet 238e56f8d83748e8b2cf1100e92db4c6 with delete_type TABLET_DATA_TOMBSTONED (TS 35f356f20e274db7bf020f78f1bc9ae7 not found in new config with opid_index 1112) from {username='slave'} at 127.0.0.1:46798
I20240502 01:55:10.919692  5653 tablet_replica.cc:330] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7: stopping tablet replica
I20240502 01:55:10.919914  5653 raft_consensus.cc:2229] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:10.920130  5653 raft_consensus.cc:2258] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:10.921440  5653 ts_tablet_manager.cc:1875] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:10.926297  5653 ts_tablet_manager.cc:1888] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.1111
I20240502 01:55:10.926424  5653 log.cc:1196] T 238e56f8d83748e8b2cf1100e92db4c6 P 35f356f20e274db7bf020f78f1bc9ae7: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal/wals/238e56f8d83748e8b2cf1100e92db4c6
I20240502 01:55:10.927142 31490 catalog_manager.cc:4830] TS 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130:41639): tablet 238e56f8d83748e8b2cf1100e92db4c6 (table rebalance_test_table_1 [id=76d2ee30f2bb480087daf717aa077ba2]) successfully deleted
I20240502 01:55:11.025167 31896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:55:11.028990 31764 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:11.041474   766 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:11.046942  4729 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:11.048472   497 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:11.060427   630 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:55:11.079080 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:11.092767   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:11.206465  4749 consensus_queue.cc:237] T 5575decad4be448e835db06b570c69a8 P 3f175b3ff87e41dba66ecb07ceb5f896 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1221, Committed index: 1221, Last appended: 3.1221, Last appended by leader: 1220, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1222 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } }
I20240502 01:55:11.208081 31647 raft_consensus.cc:1261] T 5575decad4be448e835db06b570c69a8 P b191086bf5f2432b8d240510817ae5fe [term 3 FOLLOWER]: Refusing update from remote peer 3f175b3ff87e41dba66ecb07ceb5f896: Log matching property violated. Preceding OpId in replica: term: 3 index: 1221. Preceding OpId from leader: term: 3 index: 1222. (index mismatch)
I20240502 01:55:11.208595  5650 consensus_queue.cc:1035] T 5575decad4be448e835db06b570c69a8 P 3f175b3ff87e41dba66ecb07ceb5f896 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1222, Last known committed idx: 1221, Time since last communication: 0.000s
I20240502 01:55:11.210669  5474 raft_consensus.cc:2932] T 5575decad4be448e835db06b570c69a8 P 3f175b3ff87e41dba66ecb07ceb5f896 [term 3 LEADER]: Committing config change with OpId 3.1222: config changed from index 1220 to 1222, VOTER a506969c1a5346c09c19a01f6cbc1a8b (127.25.109.133) evicted. New config: { opid_index: 1222 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } } }
I20240502 01:55:11.211182 31647 raft_consensus.cc:2932] T 5575decad4be448e835db06b570c69a8 P b191086bf5f2432b8d240510817ae5fe [term 3 FOLLOWER]: Committing config change with OpId 3.1222: config changed from index 1220 to 1222, VOTER a506969c1a5346c09c19a01f6cbc1a8b (127.25.109.133) evicted. New config: { opid_index: 1222 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } } }
I20240502 01:55:11.215713 31502 catalog_manager.cc:5484] T 5575decad4be448e835db06b570c69a8 P 3f175b3ff87e41dba66ecb07ceb5f896 reported cstate change: config changed from index 1220 to 1222, VOTER a506969c1a5346c09c19a01f6cbc1a8b (127.25.109.133) evicted. New cstate: current_term: 3 leader_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" committed_config { opid_index: 1222 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "3f175b3ff87e41dba66ecb07ceb5f896" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 45685 } } }
I20240502 01:55:11.234135   630 tablet_service.cc:1514] Processing DeleteTablet for tablet 5575decad4be448e835db06b570c69a8 with delete_type TABLET_DATA_TOMBSTONED (TS a506969c1a5346c09c19a01f6cbc1a8b not found in new config with opid_index 1222) from {username='slave'} at 127.0.0.1:41526
I20240502 01:55:11.234901  5711 tablet_replica.cc:330] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b: stopping tablet replica
I20240502 01:55:11.236675  5711 raft_consensus.cc:2229] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:11.236984  5711 raft_consensus.cc:2258] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:11.239737  5711 ts_tablet_manager.cc:1875] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:11.245138  5711 ts_tablet_manager.cc:1888] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.1221
I20240502 01:55:11.245273  5711 log.cc:1196] T 5575decad4be448e835db06b570c69a8 P a506969c1a5346c09c19a01f6cbc1a8b: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal/wals/5575decad4be448e835db06b570c69a8
I20240502 01:55:11.246168 31492 catalog_manager.cc:4830] TS a506969c1a5346c09c19a01f6cbc1a8b (127.25.109.133:46379): tablet 5575decad4be448e835db06b570c69a8 (table rebalance_test_table_3 [id=ae2e57387f5a496cb85e41e055010f5b]) successfully deleted
I20240502 01:55:11.426154 31764 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:11.446106 31896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:55:11.446583   497 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:11.450489   766 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:11.474138   630 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:55:11.477018  4729 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:11.477017 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:11.514647   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:11.735249   497 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:11.740149 31896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:55:11.742048 31764 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:11.748168  4729 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:11.755198   766 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:11.766289   630 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:55:11.775486 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:11.783365   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:12.164897   497 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:12.171068 31896 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:55:12.177102   766 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:12.192030   630 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:55:12.192735 31628 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:12.201129 31764 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:12.209034  4729 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:12.211670   899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 15725d2cffce4f5da0ed6c7449588f85 | 127.25.109.190:41417 | HEALTHY

Unusual flags for Master:
                   Flag                   |                                                                                Value                                                                                 |         Tags          |         Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 allow_unsafe_replication_factor          | true                                                                                                                                                                 | runtime,unsafe        | all 1 server(s) checked
 ipki_ca_key_size                         | 768                                                                                                                                                                  | experimental          | all 1 server(s) checked
 ipki_server_key_size                     | 768                                                                                                                                                                  | experimental          | all 1 server(s) checked
 never_fsync                              | true                                                                                                                                                                 | unsafe,advanced       | all 1 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                    | unsafe,hidden         | all 1 server(s) checked
 raft_prepare_replacement_before_eviction | false                                                                                                                                                                | experimental,advanced | all 1 server(s) checked
 rpc_reuseport                            | true                                                                                                                                                                 | experimental          | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                 | unsafe                | all 1 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                   | hidden                | all 1 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden                | all 1 server(s) checked
 tsk_num_rsa_bits                         | 512                                                                                                                                                                  | experimental          | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |        Value         |         Master
---------------------+----------------------+-------------------------
 builtin_ntp_servers | 127.25.109.148:40813 | all 1 server(s) checked
 time_source         | builtin              | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address        | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
 0aea20ce2a1144519ed144383c016d78 | 127.25.109.132:34649 | HEALTHY | <none>   |       1        |       0
 1864f9ea44c54742b090c594a5be3669 | 127.25.109.131:42119 | HEALTHY | <none>   |       3        |       0
 35f356f20e274db7bf020f78f1bc9ae7 | 127.25.109.130:41639 | HEALTHY | <none>   |       4        |       0
 3f175b3ff87e41dba66ecb07ceb5f896 | 127.25.109.136:45685 | HEALTHY | <none>   |       2        |       0
 676092c80dd74040bb49a06db1ac209c | 127.25.109.134:38633 | HEALTHY | <none>   |       1        |       0
 a506969c1a5346c09c19a01f6cbc1a8b | 127.25.109.133:46379 | HEALTHY | <none>   |       3        |       0
 b191086bf5f2432b8d240510817ae5fe | 127.25.109.129:42243 | HEALTHY | <none>   |       2        |       0
 facf1046c8054d2d841cf875c63e29e2 | 127.25.109.135:37253 | HEALTHY | <none>   |       0        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       8

Unusual flags for Tablet Server:
                   Flag                   |                                                                              Value                                                                               |         Tags          |      Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_server_key_size                     | 768                                                                                                                                                              | experimental          | all 8 server(s) checked
 local_ip_for_outbound_sockets            | 127.25.109.129                                                                                                                                                   | experimental          | 127.25.109.129:42243
 local_ip_for_outbound_sockets            | 127.25.109.130                                                                                                                                                   | experimental          | 127.25.109.130:41639
 local_ip_for_outbound_sockets            | 127.25.109.131                                                                                                                                                   | experimental          | 127.25.109.131:42119
 local_ip_for_outbound_sockets            | 127.25.109.132                                                                                                                                                   | experimental          | 127.25.109.132:34649
 local_ip_for_outbound_sockets            | 127.25.109.133                                                                                                                                                   | experimental          | 127.25.109.133:46379
 local_ip_for_outbound_sockets            | 127.25.109.134                                                                                                                                                   | experimental          | 127.25.109.134:38633
 local_ip_for_outbound_sockets            | 127.25.109.135                                                                                                                                                   | experimental          | 127.25.109.135:37253
 local_ip_for_outbound_sockets            | 127.25.109.136                                                                                                                                                   | experimental          | 127.25.109.136:45685
 never_fsync                              | true                                                                                                                                                             | unsafe,advanced       | all 8 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                | unsafe,hidden         | all 8 server(s) checked
 raft_prepare_replacement_before_eviction | false                                                                                                                                                            | experimental,advanced | all 8 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                             | unsafe                | all 8 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                               | hidden                | all 8 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden                | 127.25.109.129:42243
 server_dump_info_path                    | /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden                | 127.25.109.130:41639
 server_dump_info_path                    | /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden                | 127.25.109.131:42119
 server_dump_info_path                    | /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden                | 127.25.109.132:34649
 server_dump_info_path                    | /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden                | 127.25.109.133:46379
 server_dump_info_path                    | /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden                | 127.25.109.134:38633
 server_dump_info_path                    | /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden                | 127.25.109.135:37253
 server_dump_info_path                    | /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden                | 127.25.109.136:45685

Flags of checked categories for Tablet Server:
        Flag         |        Value         |      Tablet Server
---------------------+----------------------+-------------------------
 builtin_ntp_servers | 127.25.109.148:40813 | all 8 server(s) checked
 time_source         | builtin              | all 8 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.18.0-SNAPSHOT | all 9 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
          Name          | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
 TestTable              | 2  | HEALTHY | 1             | 1       | 0          | 0                | 0
 rebalance_test_table_0 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_1 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_2 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_3 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_4 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 4
 First Quartile | 4
 Median         | 4
 Third Quartile | 4
 Maximum        | 4

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 8
 Tables         | 6
 Tablets        | 16
 Replicas       | 32

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20240502 01:55:12.304848 26038 log_verifier.cc:126] Checking tablet 02b02a474b5e4f6ab173ced2dc21c03e
I20240502 01:55:12.736901 26038 log_verifier.cc:177] Verified matching terms for 1164 ops in tablet 02b02a474b5e4f6ab173ced2dc21c03e
I20240502 01:55:12.738183 26038 log_verifier.cc:126] Checking tablet 10aa745ed48d46a3b0558d35dee6a807
I20240502 01:55:13.094873 26038 log_verifier.cc:177] Verified matching terms for 1144 ops in tablet 10aa745ed48d46a3b0558d35dee6a807
I20240502 01:55:13.096199 26038 log_verifier.cc:126] Checking tablet 1313391ae5b2440d93445deda2642a14
I20240502 01:55:13.488204 26038 log_verifier.cc:177] Verified matching terms for 1141 ops in tablet 1313391ae5b2440d93445deda2642a14
I20240502 01:55:13.489486 26038 log_verifier.cc:126] Checking tablet 238e56f8d83748e8b2cf1100e92db4c6
I20240502 01:55:13.872277 26038 log_verifier.cc:177] Verified matching terms for 1112 ops in tablet 238e56f8d83748e8b2cf1100e92db4c6
I20240502 01:55:13.873615 26038 log_verifier.cc:126] Checking tablet 4a00f35083c9404ea2ea877826f2fe77
I20240502 01:55:14.274044 26038 log_verifier.cc:177] Verified matching terms for 1180 ops in tablet 4a00f35083c9404ea2ea877826f2fe77
I20240502 01:55:14.275368 26038 log_verifier.cc:126] Checking tablet 5575decad4be448e835db06b570c69a8
I20240502 01:55:14.635219 26038 log_verifier.cc:177] Verified matching terms for 1222 ops in tablet 5575decad4be448e835db06b570c69a8
I20240502 01:55:14.637120 26038 log_verifier.cc:126] Checking tablet 559610821917470fbcf5641d36acf409
I20240502 01:55:14.972918 26038 log_verifier.cc:177] Verified matching terms for 1054 ops in tablet 559610821917470fbcf5641d36acf409
I20240502 01:55:14.974146 26038 log_verifier.cc:126] Checking tablet 65576af2c4bb4f25b4d14963cbe65ca7
I20240502 01:55:15.314121 26038 log_verifier.cc:177] Verified matching terms for 1180 ops in tablet 65576af2c4bb4f25b4d14963cbe65ca7
I20240502 01:55:15.315490 26038 log_verifier.cc:126] Checking tablet 6b56adc64eba4898b80687a2679b70c8
I20240502 01:55:15.690241 26038 log_verifier.cc:177] Verified matching terms for 1180 ops in tablet 6b56adc64eba4898b80687a2679b70c8
I20240502 01:55:15.691614 26038 log_verifier.cc:126] Checking tablet 704700e8872b4675b41d6c172731a5e0
I20240502 01:55:16.029068 26038 log_verifier.cc:177] Verified matching terms for 1178 ops in tablet 704700e8872b4675b41d6c172731a5e0
I20240502 01:55:16.030542 26038 log_verifier.cc:126] Checking tablet 82e5853a509c43df917f257186844ddc
I20240502 01:55:16.391600 26038 log_verifier.cc:177] Verified matching terms for 1150 ops in tablet 82e5853a509c43df917f257186844ddc
I20240502 01:55:16.392894 26038 log_verifier.cc:126] Checking tablet 86f2e43af7fb4e02bb853f0748d62ddb
I20240502 01:55:16.722765 26038 log_verifier.cc:177] Verified matching terms for 1110 ops in tablet 86f2e43af7fb4e02bb853f0748d62ddb
I20240502 01:55:16.723980 26038 log_verifier.cc:126] Checking tablet a865d3891d4e4430a0a0aec06a074cea
I20240502 01:55:16.735169 26038 log_verifier.cc:177] Verified matching terms for 2 ops in tablet a865d3891d4e4430a0a0aec06a074cea
I20240502 01:55:16.735250 26038 log_verifier.cc:126] Checking tablet e714cba54201433caa218073097c1e6a
I20240502 01:55:17.109926 26038 log_verifier.cc:177] Verified matching terms for 1186 ops in tablet e714cba54201433caa218073097c1e6a
I20240502 01:55:17.111219 26038 log_verifier.cc:126] Checking tablet f2a3a225d7f343d9b0ac75e41f0d2a11
I20240502 01:55:17.425606 26038 log_verifier.cc:177] Verified matching terms for 1094 ops in tablet f2a3a225d7f343d9b0ac75e41f0d2a11
I20240502 01:55:17.426797 26038 log_verifier.cc:126] Checking tablet fe2daac9229e44c9bee5f7fbaf886728
I20240502 01:55:17.795768 26038 log_verifier.cc:177] Verified matching terms for 1111 ops in tablet fe2daac9229e44c9bee5f7fbaf886728
I20240502 01:55:17.800379 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 31565
I20240502 01:55:17.816000 31691 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:55:18.133136 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 31565
W20240502 01:55:18.181927   449 connection.cc:535] server connection from 127.25.109.129:58823 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:55:18.182060 31850 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.109.129:42243 (error 108)
W20240502 01:55:18.182740  4683 connection.cc:535] client connection to 127.25.109.129:42243 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:55:18.182817 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 31701
W20240502 01:55:18.183007  4683 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:55:18.185184 31850 consensus_peers.cc:485] T f2a3a225d7f343d9b0ac75e41f0d2a11 P 1864f9ea44c54742b090c594a5be3669 -> Peer b191086bf5f2432b8d240510817ae5fe (127.25.109.129:42243): Couldn't send request to peer b191086bf5f2432b8d240510817ae5fe. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:42243: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:18.185904  4683 consensus_peers.cc:485] T 5575decad4be448e835db06b570c69a8 P 3f175b3ff87e41dba66ecb07ceb5f896 -> Peer b191086bf5f2432b8d240510817ae5fe (127.25.109.129:42243): Couldn't send request to peer b191086bf5f2432b8d240510817ae5fe. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:42243: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:18.189527 31824 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:55:18.519579 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 31701
W20240502 01:55:18.568167   719 connection.cc:535] server connection from 127.25.109.130:36309 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:55:18.568858 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 31834
I20240502 01:55:18.574936 31955 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:55:18.906618 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 31834
W20240502 01:55:18.953883   451 connection.cc:535] server connection from 127.25.109.131:52207 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:55:18.954500 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 433
I20240502 01:55:18.960088   556 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:55:19.040244  5864 raft_consensus.cc:482] T 02b02a474b5e4f6ab173ced2dc21c03e P facf1046c8054d2d841cf875c63e29e2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b191086bf5f2432b8d240510817ae5fe)
I20240502 01:55:19.040460  5864 raft_consensus.cc:504] T 02b02a474b5e4f6ab173ced2dc21c03e P facf1046c8054d2d841cf875c63e29e2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 384 OBSOLETE_local: false peers { permanent_uuid: "b191086bf5f2432b8d240510817ae5fe" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 42243 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } }
I20240502 01:55:19.041823  5864 leader_election.cc:288] T 02b02a474b5e4f6ab173ced2dc21c03e P facf1046c8054d2d841cf875c63e29e2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b191086bf5f2432b8d240510817ae5fe (127.25.109.129:42243)
W20240502 01:55:19.042783   852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.129:42243: connect: Connection refused (error 111)
W20240502 01:55:19.044005   852 leader_election.cc:334] T 02b02a474b5e4f6ab173ced2dc21c03e P facf1046c8054d2d841cf875c63e29e2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b191086bf5f2432b8d240510817ae5fe (127.25.109.129:42243): Network error: Client connection negotiation failed: client connection to 127.25.109.129:42243: connect: Connection refused (error 111)
I20240502 01:55:19.044158   852 leader_election.cc:302] T 02b02a474b5e4f6ab173ced2dc21c03e P facf1046c8054d2d841cf875c63e29e2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: facf1046c8054d2d841cf875c63e29e2; no voters: b191086bf5f2432b8d240510817ae5fe
I20240502 01:55:19.044484  5864 raft_consensus.cc:2726] T 02b02a474b5e4f6ab173ced2dc21c03e P facf1046c8054d2d841cf875c63e29e2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:55:19.146917   581 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.131:42119: connect: Connection refused (error 111)
W20240502 01:55:19.148427   581 consensus_peers.cc:485] T a865d3891d4e4430a0a0aec06a074cea P a506969c1a5346c09c19a01f6cbc1a8b -> Peer 1864f9ea44c54742b090c594a5be3669 (127.25.109.131:42119): Couldn't send request to peer 1864f9ea44c54742b090c594a5be3669. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.131:42119: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:19.242543 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 433
W20240502 01:55:19.287570   583 connection.cc:535] server connection from 127.25.109.132:54655 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:55:19.287947 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 566
I20240502 01:55:19.294221   693 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 01:55:19.382045  4682 consensus_peers.cc:485] T 238e56f8d83748e8b2cf1100e92db4c6 P 3f175b3ff87e41dba66ecb07ceb5f896 -> Peer 0aea20ce2a1144519ed144383c016d78 (127.25.109.132:34649): Couldn't send request to peer 0aea20ce2a1144519ed144383c016d78. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.132:34649: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:19.591171  5872 raft_consensus.cc:482] T e714cba54201433caa218073097c1e6a P facf1046c8054d2d841cf875c63e29e2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 35f356f20e274db7bf020f78f1bc9ae7)
I20240502 01:55:19.591358  5872 raft_consensus.cc:504] T e714cba54201433caa218073097c1e6a P facf1046c8054d2d841cf875c63e29e2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 597 OBSOLETE_local: false peers { permanent_uuid: "35f356f20e274db7bf020f78f1bc9ae7" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 41639 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } }
I20240502 01:55:19.592051  5872 leader_election.cc:288] T e714cba54201433caa218073097c1e6a P facf1046c8054d2d841cf875c63e29e2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130:41639)
W20240502 01:55:19.595645   852 leader_election.cc:334] T e714cba54201433caa218073097c1e6a P facf1046c8054d2d841cf875c63e29e2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 35f356f20e274db7bf020f78f1bc9ae7 (127.25.109.130:41639): Network error: Client connection negotiation failed: client connection to 127.25.109.130:41639: connect: Connection refused (error 111)
I20240502 01:55:19.596303   852 leader_election.cc:302] T e714cba54201433caa218073097c1e6a P facf1046c8054d2d841cf875c63e29e2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: facf1046c8054d2d841cf875c63e29e2; no voters: 35f356f20e274db7bf020f78f1bc9ae7
I20240502 01:55:19.596633  5872 raft_consensus.cc:2726] T e714cba54201433caa218073097c1e6a P facf1046c8054d2d841cf875c63e29e2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:55:19.598759 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 566
I20240502 01:55:19.641697 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 702
I20240502 01:55:19.647547   827 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:55:19.934633  5872 raft_consensus.cc:482] T 1313391ae5b2440d93445deda2642a14 P facf1046c8054d2d841cf875c63e29e2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1864f9ea44c54742b090c594a5be3669)
I20240502 01:55:19.934875  5872 raft_consensus.cc:504] T 1313391ae5b2440d93445deda2642a14 P facf1046c8054d2d841cf875c63e29e2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 583 OBSOLETE_local: false peers { permanent_uuid: "1864f9ea44c54742b090c594a5be3669" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 42119 } } peers { permanent_uuid: "facf1046c8054d2d841cf875c63e29e2" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 37253 } }
I20240502 01:55:19.935490  5872 leader_election.cc:288] T 1313391ae5b2440d93445deda2642a14 P facf1046c8054d2d841cf875c63e29e2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1864f9ea44c54742b090c594a5be3669 (127.25.109.131:42119)
W20240502 01:55:19.936807   851 leader_election.cc:334] T 1313391ae5b2440d93445deda2642a14 P facf1046c8054d2d841cf875c63e29e2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1864f9ea44c54742b090c594a5be3669 (127.25.109.131:42119): Network error: Client connection negotiation failed: client connection to 127.25.109.131:42119: connect: Connection refused (error 111)
I20240502 01:55:19.936939   851 leader_election.cc:302] T 1313391ae5b2440d93445deda2642a14 P facf1046c8054d2d841cf875c63e29e2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: facf1046c8054d2d841cf875c63e29e2; no voters: 1864f9ea44c54742b090c594a5be3669
I20240502 01:55:19.937170  5872 raft_consensus.cc:2726] T 1313391ae5b2440d93445deda2642a14 P facf1046c8054d2d841cf875c63e29e2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:55:19.945762 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 702
I20240502 01:55:19.991917 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 836
I20240502 01:55:19.997745   960 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:55:20.276201 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 836
W20240502 01:55:20.320489 31490 connection.cc:535] server connection from 127.25.109.135:36039 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:55:20.320686 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 4667
I20240502 01:55:20.326807  4791 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:55:20.530489 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 4667
I20240502 01:55:20.559013 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 31474
I20240502 01:55:20.560350 31535 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:55:20.825343 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 31474
2024-05-02T01:55:20Z chronyd exiting
[       OK ] RebalanceParamTest.Rebalance/2 (118691 ms)
[----------] 1 test from RebalanceParamTest (118691 ms total)

[----------] 1 test from ConcurrentRebalancersTest
[ RUN      ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/0
I20240502 01:55:20.933612 26038 test_util.cc:268] Using random seed: -448139056
I20240502 01:55:20.935313 26038 ts_itest-base.cc:112] Starting cluster with:
I20240502 01:55:20.935385 26038 ts_itest-base.cc:113] --------------
I20240502 01:55:20.935431 26038 ts_itest-base.cc:114] 8 tablet servers
I20240502 01:55:20.935477 26038 ts_itest-base.cc:115] 3 replicas per TS
I20240502 01:55:20.935528 26038 ts_itest-base.cc:116] --------------
2024-05-02T01:55:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T01:55:20Z Disabled control of system clock
I20240502 01:55:20.998147 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.190:37927
--webserver_interface=127.25.109.190
--webserver_port=0
--builtin_ntp_servers=127.25.109.148:46095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.190:37927
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240502 01:55:21.352712  5890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:21.353077  5890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:21.353210  5890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:21.360970  5890 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:55:21.361136  5890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 01:55:21.361224  5890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:21.361304  5890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 01:55:21.361372  5890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 01:55:21.370950  5890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:46095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.25.109.190:37927
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.190:37927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.25.109.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:21.372426  5890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:21.374305  5890 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:21.386040  5895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:21.386410  5896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:21.386843  5898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:21.387164  5890 server_base.cc:1034] running on GCE node
I20240502 01:55:21.388082  5890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:21.389619  5890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:21.390867  5890 hybrid_clock.cc:648] HybridClock initialized: now 1714614921390802 us; error 43 us; skew 500 ppm
I20240502 01:55:21.391245  5890 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:21.393653  5890 webserver.cc:460] Webserver started at http://127.25.109.190:39405/ using document root <none> and password file <none>
I20240502 01:55:21.394414  5890 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:21.394538  5890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:21.394873  5890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:55:21.397641  5890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "ee06754b44a64b83a28dea3c44bf4dfc"
format_stamp: "Formatted at 2024-05-02 01:55:21 on dist-test-slave-kj4b"
I20240502 01:55:21.398427  5890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "ee06754b44a64b83a28dea3c44bf4dfc"
format_stamp: "Formatted at 2024-05-02 01:55:21 on dist-test-slave-kj4b"
I20240502 01:55:21.403829  5890 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.002s	sys 0.005s
I20240502 01:55:21.407301  5904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:21.408648  5890 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 01:55:21.408833  5890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal
uuid: "ee06754b44a64b83a28dea3c44bf4dfc"
format_stamp: "Formatted at 2024-05-02 01:55:21 on dist-test-slave-kj4b"
I20240502 01:55:21.409035  5890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:21.449853  5890 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:55:21.450816  5890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:21.451135  5890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:21.469601  5890 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.190:37927
I20240502 01:55:21.469635  5956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.190:37927 every 8 connection(s)
I20240502 01:55:21.471237  5890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 01:55:21.472791 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 5890
I20240502 01:55:21.473014 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 01:55:21.476485  5957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:21.488282  5957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee06754b44a64b83a28dea3c44bf4dfc: Bootstrap starting.
I20240502 01:55:21.491592  5957 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee06754b44a64b83a28dea3c44bf4dfc: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:21.492856  5957 log.cc:824] T 00000000000000000000000000000000 P ee06754b44a64b83a28dea3c44bf4dfc: Log is configured to *not* fsync() on all Append() calls
I20240502 01:55:21.495461  5957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee06754b44a64b83a28dea3c44bf4dfc: No bootstrap required, opened a new log
I20240502 01:55:21.501909  5957 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee06754b44a64b83a28dea3c44bf4dfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee06754b44a64b83a28dea3c44bf4dfc" member_type: VOTER last_known_addr { host: "127.25.109.190" port: 37927 } }
I20240502 01:55:21.502255  5957 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee06754b44a64b83a28dea3c44bf4dfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:21.502341  5957 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee06754b44a64b83a28dea3c44bf4dfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee06754b44a64b83a28dea3c44bf4dfc, State: Initialized, Role: FOLLOWER
I20240502 01:55:21.502903  5957 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee06754b44a64b83a28dea3c44bf4dfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee06754b44a64b83a28dea3c44bf4dfc" member_type: VOTER last_known_addr { host: "127.25.109.190" port: 37927 } }
I20240502 01:55:21.503093  5957 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee06754b44a64b83a28dea3c44bf4dfc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:55:21.503188  5957 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee06754b44a64b83a28dea3c44bf4dfc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:55:21.503311  5957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee06754b44a64b83a28dea3c44bf4dfc [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:21.505605  5957 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee06754b44a64b83a28dea3c44bf4dfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee06754b44a64b83a28dea3c44bf4dfc" member_type: VOTER last_known_addr { host: "127.25.109.190" port: 37927 } }
I20240502 01:55:21.506075  5957 leader_election.cc:302] T 00000000000000000000000000000000 P ee06754b44a64b83a28dea3c44bf4dfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee06754b44a64b83a28dea3c44bf4dfc; no voters: 
I20240502 01:55:21.506618  5957 leader_election.cc:288] T 00000000000000000000000000000000 P ee06754b44a64b83a28dea3c44bf4dfc [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 01:55:21.506760  5962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee06754b44a64b83a28dea3c44bf4dfc [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:21.507375  5962 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee06754b44a64b83a28dea3c44bf4dfc [term 1 LEADER]: Becoming Leader. State: Replica: ee06754b44a64b83a28dea3c44bf4dfc, State: Running, Role: LEADER
I20240502 01:55:21.507913  5962 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee06754b44a64b83a28dea3c44bf4dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee06754b44a64b83a28dea3c44bf4dfc" member_type: VOTER last_known_addr { host: "127.25.109.190" port: 37927 } }
I20240502 01:55:21.508613  5957 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee06754b44a64b83a28dea3c44bf4dfc [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:55:21.510511  5963 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee06754b44a64b83a28dea3c44bf4dfc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee06754b44a64b83a28dea3c44bf4dfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee06754b44a64b83a28dea3c44bf4dfc" member_type: VOTER last_known_addr { host: "127.25.109.190" port: 37927 } } }
I20240502 01:55:21.510887  5963 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee06754b44a64b83a28dea3c44bf4dfc [sys.catalog]: This master's current role is: LEADER
I20240502 01:55:21.511230  5964 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee06754b44a64b83a28dea3c44bf4dfc [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee06754b44a64b83a28dea3c44bf4dfc. Latest consensus state: current_term: 1 leader_uuid: "ee06754b44a64b83a28dea3c44bf4dfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee06754b44a64b83a28dea3c44bf4dfc" member_type: VOTER last_known_addr { host: "127.25.109.190" port: 37927 } } }
I20240502 01:55:21.511593  5964 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee06754b44a64b83a28dea3c44bf4dfc [sys.catalog]: This master's current role is: LEADER
I20240502 01:55:21.514485  5968 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:55:21.519625  5968 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:55:21.528700  5968 catalog_manager.cc:1347] Generated new cluster ID: 25752eaa97cf4f85a9cd76ee3cbf7c5b
I20240502 01:55:21.528821  5968 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:55:21.542614  5968 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:55:21.543790  5968 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:55:21.556432  5968 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ee06754b44a64b83a28dea3c44bf4dfc: Generated new TSK 0
I20240502 01:55:21.557587  5968 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:55:21.574659 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.129:0
--local_ip_for_outbound_sockets=127.25.109.129
--webserver_interface=127.25.109.129
--webserver_port=0
--tserver_master_addrs=127.25.109.190:37927
--builtin_ntp_servers=127.25.109.148:46095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 01:55:21.938517  5982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:21.938799  5982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:21.938920  5982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:21.946381  5982 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:55:21.946521  5982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:21.946676  5982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.129
I20240502 01:55:21.956244  5982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:46095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.25.109.129
--webserver_port=0
--tserver_master_addrs=127.25.109.190:37927
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.129
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:21.957608  5982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:21.959426  5982 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:21.972396  5990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:21.972427  5988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:21.972872  5987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:21.973502  5982 server_base.cc:1034] running on GCE node
I20240502 01:55:21.974139  5982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:21.975152  5982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:21.976364  5982 hybrid_clock.cc:648] HybridClock initialized: now 1714614921976313 us; error 53 us; skew 500 ppm
I20240502 01:55:21.976723  5982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:21.979135  5982 webserver.cc:460] Webserver started at http://127.25.109.129:38457/ using document root <none> and password file <none>
I20240502 01:55:21.979967  5982 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:21.980093  5982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:21.980429  5982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:55:21.983181  5982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "218bc8c1f9ce46059898fc738179adf9"
format_stamp: "Formatted at 2024-05-02 01:55:21 on dist-test-slave-kj4b"
I20240502 01:55:21.984006  5982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "218bc8c1f9ce46059898fc738179adf9"
format_stamp: "Formatted at 2024-05-02 01:55:21 on dist-test-slave-kj4b"
I20240502 01:55:21.989655  5982 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20240502 01:55:21.993184  5997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:21.994597  5982 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240502 01:55:21.994763  5982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "218bc8c1f9ce46059898fc738179adf9"
format_stamp: "Formatted at 2024-05-02 01:55:21 on dist-test-slave-kj4b"
I20240502 01:55:21.994971  5982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:22.008034  5982 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:55:22.008970  5982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:22.009238  5982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:22.010250  5982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:22.011827  5982 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:22.011952  5982 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.001s	sys 0.000s
I20240502 01:55:22.012065  5982 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:22.012140  5982 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:22.044499  5982 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.129:36227
I20240502 01:55:22.044571  6114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.129:36227 every 8 connection(s)
I20240502 01:55:22.045888  5982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 01:55:22.047323 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 5982
I20240502 01:55:22.047566 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 01:55:22.051460 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.130:0
--local_ip_for_outbound_sockets=127.25.109.130
--webserver_interface=127.25.109.130
--webserver_port=0
--tserver_master_addrs=127.25.109.190:37927
--builtin_ntp_servers=127.25.109.148:46095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:55:22.060214  6115 heartbeater.cc:347] Connected to a master server at 127.25.109.190:37927
I20240502 01:55:22.060562  6115 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:22.061378  6115 heartbeater.cc:511] Master 127.25.109.190:37927 requested a full tablet report, sending...
I20240502 01:55:22.063558  5922 ts_manager.cc:194] Registered new tserver with Master: 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227)
I20240502 01:55:22.065181  5922 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.129:40579
W20240502 01:55:22.398602  6119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:22.398903  6119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:22.399015  6119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:22.406797  6119 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:55:22.406927  6119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:22.407050  6119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.130
I20240502 01:55:22.416528  6119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:46095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.25.109.130
--webserver_port=0
--tserver_master_addrs=127.25.109.190:37927
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.130
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:22.417866  6119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:22.419533  6119 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:22.431514  6127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:22.431492  6125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:22.432032  6124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:22.432601  6119 server_base.cc:1034] running on GCE node
I20240502 01:55:22.433187  6119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:22.434145  6119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:22.435343  6119 hybrid_clock.cc:648] HybridClock initialized: now 1714614922435289 us; error 42 us; skew 500 ppm
I20240502 01:55:22.435652  6119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:22.437857  6119 webserver.cc:460] Webserver started at http://127.25.109.130:35417/ using document root <none> and password file <none>
I20240502 01:55:22.438571  6119 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:22.438685  6119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:22.439008  6119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:55:22.441642  6119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "a2c90ee440b44b708bccc2e2d5385cc0"
format_stamp: "Formatted at 2024-05-02 01:55:22 on dist-test-slave-kj4b"
I20240502 01:55:22.442391  6119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "a2c90ee440b44b708bccc2e2d5385cc0"
format_stamp: "Formatted at 2024-05-02 01:55:22 on dist-test-slave-kj4b"
I20240502 01:55:22.447518  6119 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.002s	sys 0.005s
I20240502 01:55:22.451023  6133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:22.452554  6119 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.003s
I20240502 01:55:22.452725  6119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "a2c90ee440b44b708bccc2e2d5385cc0"
format_stamp: "Formatted at 2024-05-02 01:55:22 on dist-test-slave-kj4b"
I20240502 01:55:22.452941  6119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:22.486918  6119 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:55:22.487895  6119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:22.488206  6119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:22.489187  6119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:22.490641  6119 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:22.490727  6119 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:22.490818  6119 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:22.490872  6119 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:22.520790  6119 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.130:38197
I20240502 01:55:22.520843  6249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.130:38197 every 8 connection(s)
I20240502 01:55:22.522080  6119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 01:55:22.528985 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 6119
I20240502 01:55:22.529201 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 01:55:22.532502 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.131:0
--local_ip_for_outbound_sockets=127.25.109.131
--webserver_interface=127.25.109.131
--webserver_port=0
--tserver_master_addrs=127.25.109.190:37927
--builtin_ntp_servers=127.25.109.148:46095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:55:22.534569  6250 heartbeater.cc:347] Connected to a master server at 127.25.109.190:37927
I20240502 01:55:22.534919  6250 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:22.535781  6250 heartbeater.cc:511] Master 127.25.109.190:37927 requested a full tablet report, sending...
I20240502 01:55:22.537125  5922 ts_manager.cc:194] Registered new tserver with Master: a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197)
I20240502 01:55:22.537881  5922 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.130:39451
W20240502 01:55:22.882155  6253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:22.882484  6253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:22.882615  6253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:22.890399  6253 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:55:22.890558  6253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:22.890727  6253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.131
I20240502 01:55:22.900199  6253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:46095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.25.109.131
--webserver_port=0
--tserver_master_addrs=127.25.109.190:37927
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.131
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:22.901566  6253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:22.903348  6253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:22.915469  6261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:22.916113  6259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:22.916155  6258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:22.916599  6253 server_base.cc:1034] running on GCE node
I20240502 01:55:22.917186  6253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:22.918092  6253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:22.919284  6253 hybrid_clock.cc:648] HybridClock initialized: now 1714614922919234 us; error 44 us; skew 500 ppm
I20240502 01:55:22.919588  6253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:22.925720  6253 webserver.cc:460] Webserver started at http://127.25.109.131:39837/ using document root <none> and password file <none>
I20240502 01:55:22.926479  6253 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:22.926599  6253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:22.926924  6253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:55:22.929618  6253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "94ffd7aaf85b4e17881241b8396fa2f3"
format_stamp: "Formatted at 2024-05-02 01:55:22 on dist-test-slave-kj4b"
I20240502 01:55:22.930367  6253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "94ffd7aaf85b4e17881241b8396fa2f3"
format_stamp: "Formatted at 2024-05-02 01:55:22 on dist-test-slave-kj4b"
I20240502 01:55:22.935585  6253 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.001s
I20240502 01:55:22.938843  6267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:22.940135  6253 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 01:55:22.940294  6253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "94ffd7aaf85b4e17881241b8396fa2f3"
format_stamp: "Formatted at 2024-05-02 01:55:22 on dist-test-slave-kj4b"
I20240502 01:55:22.940485  6253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:22.953908  6253 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:55:22.954805  6253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:22.955101  6253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:22.956156  6253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:22.957769  6253 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:22.957863  6253 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:22.957971  6253 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:22.958036  6253 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:22.989418  6253 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.131:44245
I20240502 01:55:22.989475  6381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.131:44245 every 8 connection(s)
I20240502 01:55:22.990736  6253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 01:55:22.998313 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 6253
I20240502 01:55:22.998497 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 01:55:23.001912 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.132:0
--local_ip_for_outbound_sockets=127.25.109.132
--webserver_interface=127.25.109.132
--webserver_port=0
--tserver_master_addrs=127.25.109.190:37927
--builtin_ntp_servers=127.25.109.148:46095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:55:23.003255  6382 heartbeater.cc:347] Connected to a master server at 127.25.109.190:37927
I20240502 01:55:23.003562  6382 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:23.004359  6382 heartbeater.cc:511] Master 127.25.109.190:37927 requested a full tablet report, sending...
I20240502 01:55:23.005709  5922 ts_manager.cc:194] Registered new tserver with Master: 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245)
I20240502 01:55:23.006528  5922 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.131:40103
I20240502 01:55:23.068507  6115 heartbeater.cc:503] Master 127.25.109.190:37927 was elected leader, sending a full tablet report...
W20240502 01:55:23.351015  6385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:23.351325  6385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:23.351419  6385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:23.359186  6385 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:55:23.359333  6385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:23.359463  6385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.132
I20240502 01:55:23.368913  6385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:46095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.25.109.132
--webserver_port=0
--tserver_master_addrs=127.25.109.190:37927
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.132
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:23.370277  6385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:23.372074  6385 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:23.384285  6390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:23.384826  6391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:23.385288  6393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:23.385620  6385 server_base.cc:1034] running on GCE node
I20240502 01:55:23.386253  6385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:23.387252  6385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:23.388443  6385 hybrid_clock.cc:648] HybridClock initialized: now 1714614923388395 us; error 40 us; skew 500 ppm
I20240502 01:55:23.388772  6385 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:23.391116  6385 webserver.cc:460] Webserver started at http://127.25.109.132:38011/ using document root <none> and password file <none>
I20240502 01:55:23.391909  6385 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:23.392026  6385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:23.392346  6385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:55:23.394985  6385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "07cdb213c77c4ffda1f999b66b725399"
format_stamp: "Formatted at 2024-05-02 01:55:23 on dist-test-slave-kj4b"
I20240502 01:55:23.395701  6385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "07cdb213c77c4ffda1f999b66b725399"
format_stamp: "Formatted at 2024-05-02 01:55:23 on dist-test-slave-kj4b"
I20240502 01:55:23.400909  6385 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.001s	sys 0.003s
I20240502 01:55:23.404423  6399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:23.405776  6385 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.002s
I20240502 01:55:23.405929  6385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "07cdb213c77c4ffda1f999b66b725399"
format_stamp: "Formatted at 2024-05-02 01:55:23 on dist-test-slave-kj4b"
I20240502 01:55:23.406129  6385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:23.427664  6385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:55:23.428615  6385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:23.428901  6385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:23.429873  6385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:23.431377  6385 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:23.431474  6385 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:23.431584  6385 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:23.431655  6385 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:23.462581  6385 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.132:36867
I20240502 01:55:23.462635  6513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.132:36867 every 8 connection(s)
I20240502 01:55:23.464015  6385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 01:55:23.467407 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 6385
I20240502 01:55:23.467615 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 01:55:23.471410 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.133:0
--local_ip_for_outbound_sockets=127.25.109.133
--webserver_interface=127.25.109.133
--webserver_port=0
--tserver_master_addrs=127.25.109.190:37927
--builtin_ntp_servers=127.25.109.148:46095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:55:23.479032  6514 heartbeater.cc:347] Connected to a master server at 127.25.109.190:37927
I20240502 01:55:23.479367  6514 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:23.480250  6514 heartbeater.cc:511] Master 127.25.109.190:37927 requested a full tablet report, sending...
I20240502 01:55:23.481762  5922 ts_manager.cc:194] Registered new tserver with Master: 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867)
I20240502 01:55:23.482594  5922 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.132:39769
I20240502 01:55:23.540161  6250 heartbeater.cc:503] Master 127.25.109.190:37927 was elected leader, sending a full tablet report...
W20240502 01:55:23.826800  6517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:23.827149  6517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:23.827256  6517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:23.834784  6517 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:55:23.834909  6517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:23.835028  6517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.133
I20240502 01:55:23.844156  6517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:46095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.25.109.133
--webserver_port=0
--tserver_master_addrs=127.25.109.190:37927
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.133
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:23.845504  6517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:23.847169  6517 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:23.860338  6526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:23.860703  6523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:23.860921  6524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:23.861371  6517 server_base.cc:1034] running on GCE node
I20240502 01:55:23.862030  6517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:23.863060  6517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:23.864262  6517 hybrid_clock.cc:648] HybridClock initialized: now 1714614923864205 us; error 52 us; skew 500 ppm
I20240502 01:55:23.864622  6517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:23.867050  6517 webserver.cc:460] Webserver started at http://127.25.109.133:39661/ using document root <none> and password file <none>
I20240502 01:55:23.867959  6517 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:23.868080  6517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:23.868425  6517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:55:23.871230  6517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "60298ffea1dd4c25a38765c2ec9ec9d5"
format_stamp: "Formatted at 2024-05-02 01:55:23 on dist-test-slave-kj4b"
I20240502 01:55:23.872085  6517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "60298ffea1dd4c25a38765c2ec9ec9d5"
format_stamp: "Formatted at 2024-05-02 01:55:23 on dist-test-slave-kj4b"
I20240502 01:55:23.877750  6517 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20240502 01:55:23.881911  6532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:23.883533  6517 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:55:23.883715  6517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "60298ffea1dd4c25a38765c2ec9ec9d5"
format_stamp: "Formatted at 2024-05-02 01:55:23 on dist-test-slave-kj4b"
I20240502 01:55:23.883973  6517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:23.907595  6517 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:55:23.908531  6517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:23.908795  6517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:23.909767  6517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:23.911347  6517 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:23.911442  6517 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:23.911526  6517 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:23.911578  6517 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:23.941807  6517 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.133:43379
I20240502 01:55:23.941866  6648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.133:43379 every 8 connection(s)
I20240502 01:55:23.943099  6517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:55:23.949613 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 6517
I20240502 01:55:23.949832 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 01:55:23.953747 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.134:0
--local_ip_for_outbound_sockets=127.25.109.134
--webserver_interface=127.25.109.134
--webserver_port=0
--tserver_master_addrs=127.25.109.190:37927
--builtin_ntp_servers=127.25.109.148:46095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:55:23.957435  6649 heartbeater.cc:347] Connected to a master server at 127.25.109.190:37927
I20240502 01:55:23.957788  6649 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:23.958739  6649 heartbeater.cc:511] Master 127.25.109.190:37927 requested a full tablet report, sending...
I20240502 01:55:23.960266  5922 ts_manager.cc:194] Registered new tserver with Master: 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133:43379)
I20240502 01:55:23.961091  5922 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.133:36489
I20240502 01:55:24.009049  6382 heartbeater.cc:503] Master 127.25.109.190:37927 was elected leader, sending a full tablet report...
W20240502 01:55:24.309283  6652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:24.309640  6652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:24.309770  6652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:24.317570  6652 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:55:24.317698  6652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:24.317862  6652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.134
I20240502 01:55:24.326939  6652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:46095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.25.109.134
--webserver_port=0
--tserver_master_addrs=127.25.109.190:37927
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.134
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:24.328302  6652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:24.330090  6652 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:24.341457  6657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:24.342079  6658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:24.342639  6660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:24.342939  6652 server_base.cc:1034] running on GCE node
I20240502 01:55:24.343498  6652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:24.344456  6652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:24.345671  6652 hybrid_clock.cc:648] HybridClock initialized: now 1714614924345598 us; error 49 us; skew 500 ppm
I20240502 01:55:24.345996  6652 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:24.348191  6652 webserver.cc:460] Webserver started at http://127.25.109.134:40249/ using document root <none> and password file <none>
I20240502 01:55:24.348915  6652 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:24.349022  6652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:24.349339  6652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:55:24.352175  6652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "9db17f84d37944098cbeb9e0d8456efb"
format_stamp: "Formatted at 2024-05-02 01:55:24 on dist-test-slave-kj4b"
I20240502 01:55:24.352985  6652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "9db17f84d37944098cbeb9e0d8456efb"
format_stamp: "Formatted at 2024-05-02 01:55:24 on dist-test-slave-kj4b"
I20240502 01:55:24.358870  6652 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.005s	sys 0.000s
I20240502 01:55:24.362870  6666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:24.364503  6652 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 01:55:24.364707  6652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "9db17f84d37944098cbeb9e0d8456efb"
format_stamp: "Formatted at 2024-05-02 01:55:24 on dist-test-slave-kj4b"
I20240502 01:55:24.364939  6652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:24.380852  6652 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:55:24.381943  6652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:24.382202  6652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:24.383260  6652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:24.385129  6652 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:24.385232  6652 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:24.385324  6652 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:24.385381  6652 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:24.416865  6652 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.134:42517
I20240502 01:55:24.416914  6779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.134:42517 every 8 connection(s)
I20240502 01:55:24.418195  6652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 01:55:24.421773 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 6652
I20240502 01:55:24.421949 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 01:55:24.425199 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.135:0
--local_ip_for_outbound_sockets=127.25.109.135
--webserver_interface=127.25.109.135
--webserver_port=0
--tserver_master_addrs=127.25.109.190:37927
--builtin_ntp_servers=127.25.109.148:46095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:55:24.430990  6780 heartbeater.cc:347] Connected to a master server at 127.25.109.190:37927
I20240502 01:55:24.431341  6780 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:24.432271  6780 heartbeater.cc:511] Master 127.25.109.190:37927 requested a full tablet report, sending...
I20240502 01:55:24.433739  5922 ts_manager.cc:194] Registered new tserver with Master: 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
I20240502 01:55:24.434625  5922 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.134:54669
I20240502 01:55:24.485304  6514 heartbeater.cc:503] Master 127.25.109.190:37927 was elected leader, sending a full tablet report...
W20240502 01:55:24.785339  6783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:24.785652  6783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:24.785753  6783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:24.793510  6783 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:55:24.793648  6783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:24.793771  6783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.135
I20240502 01:55:24.803136  6783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:46095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.25.109.135
--webserver_port=0
--tserver_master_addrs=127.25.109.190:37927
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.135
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:24.804584  6783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:24.806352  6783 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:24.819084  6791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:24.819159  6789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:24.819690  6793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:24.820231  6783 server_base.cc:1034] running on GCE node
I20240502 01:55:24.820854  6783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:24.821817  6783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:24.823029  6783 hybrid_clock.cc:648] HybridClock initialized: now 1714614924822975 us; error 46 us; skew 500 ppm
I20240502 01:55:24.823344  6783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:24.825601  6783 webserver.cc:460] Webserver started at http://127.25.109.135:45803/ using document root <none> and password file <none>
I20240502 01:55:24.826326  6783 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:24.826436  6783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:24.826738  6783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:55:24.832751  6783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "436822a397664c54b8d1b9eb3bc6cff4"
format_stamp: "Formatted at 2024-05-02 01:55:24 on dist-test-slave-kj4b"
I20240502 01:55:24.833518  6783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "436822a397664c54b8d1b9eb3bc6cff4"
format_stamp: "Formatted at 2024-05-02 01:55:24 on dist-test-slave-kj4b"
I20240502 01:55:24.839056  6783 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.002s
I20240502 01:55:24.842396  6799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:24.843647  6783 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:55:24.843806  6783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "436822a397664c54b8d1b9eb3bc6cff4"
format_stamp: "Formatted at 2024-05-02 01:55:24 on dist-test-slave-kj4b"
I20240502 01:55:24.844051  6783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:24.867101  6783 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:55:24.868108  6783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:24.868392  6783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:24.869482  6783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:24.871192  6783 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:24.871301  6783 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:24.871413  6783 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:24.871480  6783 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:24.902411  6783 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.135:40543
I20240502 01:55:24.902456  6913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.135:40543 every 8 connection(s)
I20240502 01:55:24.903708  6783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 01:55:24.906060 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 6783
I20240502 01:55:24.906281 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 01:55:24.909659 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.136:0
--local_ip_for_outbound_sockets=127.25.109.136
--webserver_interface=127.25.109.136
--webserver_port=0
--tserver_master_addrs=127.25.109.190:37927
--builtin_ntp_servers=127.25.109.148:46095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:55:24.917387  6914 heartbeater.cc:347] Connected to a master server at 127.25.109.190:37927
I20240502 01:55:24.917740  6914 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:24.918664  6914 heartbeater.cc:511] Master 127.25.109.190:37927 requested a full tablet report, sending...
I20240502 01:55:24.920168  5922 ts_manager.cc:194] Registered new tserver with Master: 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135:40543)
I20240502 01:55:24.920964  5922 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.135:42017
I20240502 01:55:24.963716  6649 heartbeater.cc:503] Master 127.25.109.190:37927 was elected leader, sending a full tablet report...
W20240502 01:55:25.266170  6917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:25.266489  6917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:25.266592  6917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:25.274598  6917 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:55:25.274744  6917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:25.274873  6917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.136
I20240502 01:55:25.284400  6917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:46095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.25.109.136
--webserver_port=0
--tserver_master_addrs=127.25.109.190:37927
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.136
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:25.285730  6917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:25.287513  6917 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:25.299607  6922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:25.300292  6925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:25.300827  6923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:25.301120  6917 server_base.cc:1034] running on GCE node
I20240502 01:55:25.301687  6917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:25.302665  6917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:25.303891  6917 hybrid_clock.cc:648] HybridClock initialized: now 1714614925303810 us; error 44 us; skew 500 ppm
I20240502 01:55:25.304225  6917 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:25.306475  6917 webserver.cc:460] Webserver started at http://127.25.109.136:34789/ using document root <none> and password file <none>
I20240502 01:55:25.307265  6917 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:25.307377  6917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:25.307698  6917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:55:25.310547  6917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "6d34e50fb5dd4a1db58013d4863b8992"
format_stamp: "Formatted at 2024-05-02 01:55:25 on dist-test-slave-kj4b"
I20240502 01:55:25.311358  6917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "6d34e50fb5dd4a1db58013d4863b8992"
format_stamp: "Formatted at 2024-05-02 01:55:25 on dist-test-slave-kj4b"
I20240502 01:55:25.316989  6917 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.002s
I20240502 01:55:25.320463  6932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:25.321815  6917 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240502 01:55:25.321972  6917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "6d34e50fb5dd4a1db58013d4863b8992"
format_stamp: "Formatted at 2024-05-02 01:55:25 on dist-test-slave-kj4b"
I20240502 01:55:25.322175  6917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:25.334389  6917 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:55:25.335240  6917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:25.335505  6917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:25.336474  6917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:25.337982  6917 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:25.338079  6917 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:25.338187  6917 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:25.338254  6917 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:25.369382  6917 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.136:44359
I20240502 01:55:25.369459  7046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.136:44359 every 8 connection(s)
I20240502 01:55:25.370857  6917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 01:55:25.376310 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 6917
I20240502 01:55:25.376482 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 01:55:25.384804  7047 heartbeater.cc:347] Connected to a master server at 127.25.109.190:37927
I20240502 01:55:25.385150  7047 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:25.385982  7047 heartbeater.cc:511] Master 127.25.109.190:37927 requested a full tablet report, sending...
I20240502 01:55:25.387465  5922 ts_manager.cc:194] Registered new tserver with Master: 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359)
I20240502 01:55:25.388383  5922 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.136:51947
I20240502 01:55:25.393308 26038 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 01:55:25.409775 26038 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 01:55:25.409968 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 6517
I20240502 01:55:25.418728  6644 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:55:25.442878  6780 heartbeater.cc:503] Master 127.25.109.190:37927 was elected leader, sending a full tablet report...
I20240502 01:55:25.506276 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 6517
I20240502 01:55:25.522775 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 6652
I20240502 01:55:25.531186  6775 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:55:25.613890 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 6652
I20240502 01:55:25.628460 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 6783
I20240502 01:55:25.636458  6909 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:55:25.717823 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 6783
I20240502 01:55:25.732045 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 6917
I20240502 01:55:25.739892  7042 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:55:25.823527 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 6917
I20240502 01:55:29.590075  5922 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58156:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:55:29.650744  6047 tablet_service.cc:1467] Processing CreateTablet for tablet 9a5f486852554e2fa2d07f7b9d686ced (DEFAULT_TABLE table=rebalance_test_table_0 [id=ef6f7a2e933049e5982deec12e9a670a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:29.652485  6448 tablet_service.cc:1467] Processing CreateTablet for tablet 086509f1ee0a4a81b2f8194b50dbe159 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ef6f7a2e933049e5982deec12e9a670a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:29.653151  6047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a5f486852554e2fa2d07f7b9d686ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:29.653844  6448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 086509f1ee0a4a81b2f8194b50dbe159. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:29.651928  6048 tablet_service.cc:1467] Processing CreateTablet for tablet 086509f1ee0a4a81b2f8194b50dbe159 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ef6f7a2e933049e5982deec12e9a670a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:29.654773  6046 tablet_service.cc:1467] Processing CreateTablet for tablet 0cecd1b14e474dee825fae040ce1b31e (DEFAULT_TABLE table=rebalance_test_table_0 [id=ef6f7a2e933049e5982deec12e9a670a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:29.655404  6048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 086509f1ee0a4a81b2f8194b50dbe159. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:29.654364  6447 tablet_service.cc:1467] Processing CreateTablet for tablet 0cecd1b14e474dee825fae040ce1b31e (DEFAULT_TABLE table=rebalance_test_table_0 [id=ef6f7a2e933049e5982deec12e9a670a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:29.657704  6447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cecd1b14e474dee825fae040ce1b31e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:29.657725  6046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cecd1b14e474dee825fae040ce1b31e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:29.658573  6316 tablet_service.cc:1467] Processing CreateTablet for tablet 9a5f486852554e2fa2d07f7b9d686ced (DEFAULT_TABLE table=rebalance_test_table_0 [id=ef6f7a2e933049e5982deec12e9a670a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:29.663429  6316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a5f486852554e2fa2d07f7b9d686ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:29.663671  6181 tablet_service.cc:1467] Processing CreateTablet for tablet 9a5f486852554e2fa2d07f7b9d686ced (DEFAULT_TABLE table=rebalance_test_table_0 [id=ef6f7a2e933049e5982deec12e9a670a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:29.661756  6315 tablet_service.cc:1467] Processing CreateTablet for tablet 0cecd1b14e474dee825fae040ce1b31e (DEFAULT_TABLE table=rebalance_test_table_0 [id=ef6f7a2e933049e5982deec12e9a670a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:29.664945  6315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cecd1b14e474dee825fae040ce1b31e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:29.665542  6181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a5f486852554e2fa2d07f7b9d686ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:29.664175  6182 tablet_service.cc:1467] Processing CreateTablet for tablet 086509f1ee0a4a81b2f8194b50dbe159 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ef6f7a2e933049e5982deec12e9a670a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:29.666064  6182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 086509f1ee0a4a81b2f8194b50dbe159. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:29.685982  7077 tablet_bootstrap.cc:492] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3: Bootstrap starting.
I20240502 01:55:29.691045  7077 tablet_bootstrap.cc:654] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:29.692082  7078 tablet_bootstrap.cc:492] T 9a5f486852554e2fa2d07f7b9d686ced P a2c90ee440b44b708bccc2e2d5385cc0: Bootstrap starting.
I20240502 01:55:29.696166  7078 tablet_bootstrap.cc:654] T 9a5f486852554e2fa2d07f7b9d686ced P a2c90ee440b44b708bccc2e2d5385cc0: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:29.696427  7079 tablet_bootstrap.cc:492] T 086509f1ee0a4a81b2f8194b50dbe159 P 07cdb213c77c4ffda1f999b66b725399: Bootstrap starting.
I20240502 01:55:29.698105  7077 log.cc:824] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3: Log is configured to *not* fsync() on all Append() calls
I20240502 01:55:29.699697  7078 log.cc:824] T 9a5f486852554e2fa2d07f7b9d686ced P a2c90ee440b44b708bccc2e2d5385cc0: Log is configured to *not* fsync() on all Append() calls
I20240502 01:55:29.701771  7079 tablet_bootstrap.cc:654] T 086509f1ee0a4a81b2f8194b50dbe159 P 07cdb213c77c4ffda1f999b66b725399: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:29.702922  7076 tablet_bootstrap.cc:492] T 086509f1ee0a4a81b2f8194b50dbe159 P 218bc8c1f9ce46059898fc738179adf9: Bootstrap starting.
I20240502 01:55:29.703266  7079 log.cc:824] T 086509f1ee0a4a81b2f8194b50dbe159 P 07cdb213c77c4ffda1f999b66b725399: Log is configured to *not* fsync() on all Append() calls
I20240502 01:55:29.706527  7079 tablet_bootstrap.cc:492] T 086509f1ee0a4a81b2f8194b50dbe159 P 07cdb213c77c4ffda1f999b66b725399: No bootstrap required, opened a new log
I20240502 01:55:29.706776  7076 tablet_bootstrap.cc:654] T 086509f1ee0a4a81b2f8194b50dbe159 P 218bc8c1f9ce46059898fc738179adf9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:29.706861  7079 ts_tablet_manager.cc:1367] T 086509f1ee0a4a81b2f8194b50dbe159 P 07cdb213c77c4ffda1f999b66b725399: Time spent bootstrapping tablet: real 0.011s	user 0.003s	sys 0.006s
I20240502 01:55:29.708156  7076 log.cc:824] T 086509f1ee0a4a81b2f8194b50dbe159 P 218bc8c1f9ce46059898fc738179adf9: Log is configured to *not* fsync() on all Append() calls
I20240502 01:55:29.709431  7078 tablet_bootstrap.cc:492] T 9a5f486852554e2fa2d07f7b9d686ced P a2c90ee440b44b708bccc2e2d5385cc0: No bootstrap required, opened a new log
I20240502 01:55:29.709942  7078 ts_tablet_manager.cc:1367] T 9a5f486852554e2fa2d07f7b9d686ced P a2c90ee440b44b708bccc2e2d5385cc0: Time spent bootstrapping tablet: real 0.018s	user 0.006s	sys 0.004s
I20240502 01:55:29.711162  7077 tablet_bootstrap.cc:492] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3: No bootstrap required, opened a new log
I20240502 01:55:29.711516  7077 ts_tablet_manager.cc:1367] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3: Time spent bootstrapping tablet: real 0.026s	user 0.006s	sys 0.006s
I20240502 01:55:29.711588  7076 tablet_bootstrap.cc:492] T 086509f1ee0a4a81b2f8194b50dbe159 P 218bc8c1f9ce46059898fc738179adf9: No bootstrap required, opened a new log
I20240502 01:55:29.711944  7076 ts_tablet_manager.cc:1367] T 086509f1ee0a4a81b2f8194b50dbe159 P 218bc8c1f9ce46059898fc738179adf9: Time spent bootstrapping tablet: real 0.009s	user 0.009s	sys 0.000s
I20240502 01:55:29.716169  7079 raft_consensus.cc:348] T 086509f1ee0a4a81b2f8194b50dbe159 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 01:55:29.716693  7079 raft_consensus.cc:374] T 086509f1ee0a4a81b2f8194b50dbe159 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:29.716836  7079 raft_consensus.cc:729] T 086509f1ee0a4a81b2f8194b50dbe159 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07cdb213c77c4ffda1f999b66b725399, State: Initialized, Role: FOLLOWER
I20240502 01:55:29.717578  7079 consensus_queue.cc:260] T 086509f1ee0a4a81b2f8194b50dbe159 P 07cdb213c77c4ffda1f999b66b725399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 01:55:29.720194  7078 raft_consensus.cc:348] T 9a5f486852554e2fa2d07f7b9d686ced P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:29.720712  7078 raft_consensus.cc:374] T 9a5f486852554e2fa2d07f7b9d686ced P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:29.720489  7077 raft_consensus.cc:348] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:29.720849  7078 raft_consensus.cc:729] T 9a5f486852554e2fa2d07f7b9d686ced P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c90ee440b44b708bccc2e2d5385cc0, State: Initialized, Role: FOLLOWER
I20240502 01:55:29.720993  7077 raft_consensus.cc:374] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:29.721129  7077 raft_consensus.cc:729] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94ffd7aaf85b4e17881241b8396fa2f3, State: Initialized, Role: FOLLOWER
I20240502 01:55:29.721792  7078 consensus_queue.cc:260] T 9a5f486852554e2fa2d07f7b9d686ced P a2c90ee440b44b708bccc2e2d5385cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:29.721961  7077 consensus_queue.cc:260] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:29.721915  7076 raft_consensus.cc:348] T 086509f1ee0a4a81b2f8194b50dbe159 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 01:55:29.722348  7076 raft_consensus.cc:374] T 086509f1ee0a4a81b2f8194b50dbe159 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:29.722510  7076 raft_consensus.cc:729] T 086509f1ee0a4a81b2f8194b50dbe159 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218bc8c1f9ce46059898fc738179adf9, State: Initialized, Role: FOLLOWER
I20240502 01:55:29.723323  7076 consensus_queue.cc:260] T 086509f1ee0a4a81b2f8194b50dbe159 P 218bc8c1f9ce46059898fc738179adf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 01:55:29.726701  7078 ts_tablet_manager.cc:1398] T 9a5f486852554e2fa2d07f7b9d686ced P a2c90ee440b44b708bccc2e2d5385cc0: Time spent starting tablet: real 0.017s	user 0.013s	sys 0.000s
I20240502 01:55:29.727274  7078 tablet_bootstrap.cc:492] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0: Bootstrap starting.
I20240502 01:55:29.728523  7077 ts_tablet_manager.cc:1398] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3: Time spent starting tablet: real 0.017s	user 0.012s	sys 0.002s
I20240502 01:55:29.728976  7077 tablet_bootstrap.cc:492] T 0cecd1b14e474dee825fae040ce1b31e P 94ffd7aaf85b4e17881241b8396fa2f3: Bootstrap starting.
I20240502 01:55:29.729771  7079 ts_tablet_manager.cc:1398] T 086509f1ee0a4a81b2f8194b50dbe159 P 07cdb213c77c4ffda1f999b66b725399: Time spent starting tablet: real 0.023s	user 0.014s	sys 0.004s
I20240502 01:55:29.730350  7079 tablet_bootstrap.cc:492] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399: Bootstrap starting.
I20240502 01:55:29.730687  7077 tablet_bootstrap.cc:654] T 0cecd1b14e474dee825fae040ce1b31e P 94ffd7aaf85b4e17881241b8396fa2f3: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:29.732148  7079 tablet_bootstrap.cc:654] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:29.735298  7079 tablet_bootstrap.cc:492] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399: No bootstrap required, opened a new log
I20240502 01:55:29.735469  7079 ts_tablet_manager.cc:1367] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399: Time spent bootstrapping tablet: real 0.005s	user 0.000s	sys 0.003s
I20240502 01:55:29.734683  7078 tablet_bootstrap.cc:654] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:29.736560  7079 raft_consensus.cc:348] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:29.736872  7079 raft_consensus.cc:374] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:29.736984  7079 raft_consensus.cc:729] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07cdb213c77c4ffda1f999b66b725399, State: Initialized, Role: FOLLOWER
I20240502 01:55:29.737250  7079 consensus_queue.cc:260] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:29.737833  7079 ts_tablet_manager.cc:1398] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 01:55:29.738435  7077 tablet_bootstrap.cc:492] T 0cecd1b14e474dee825fae040ce1b31e P 94ffd7aaf85b4e17881241b8396fa2f3: No bootstrap required, opened a new log
I20240502 01:55:29.738636  7077 ts_tablet_manager.cc:1367] T 0cecd1b14e474dee825fae040ce1b31e P 94ffd7aaf85b4e17881241b8396fa2f3: Time spent bootstrapping tablet: real 0.010s	user 0.005s	sys 0.000s
I20240502 01:55:29.739187  7078 tablet_bootstrap.cc:492] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0: No bootstrap required, opened a new log
I20240502 01:55:29.739387  7078 ts_tablet_manager.cc:1367] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0: Time spent bootstrapping tablet: real 0.012s	user 0.004s	sys 0.000s
I20240502 01:55:29.739660  7077 raft_consensus.cc:348] T 0cecd1b14e474dee825fae040ce1b31e P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:29.739957  7077 raft_consensus.cc:374] T 0cecd1b14e474dee825fae040ce1b31e P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:29.740069  7077 raft_consensus.cc:729] T 0cecd1b14e474dee825fae040ce1b31e P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94ffd7aaf85b4e17881241b8396fa2f3, State: Initialized, Role: FOLLOWER
I20240502 01:55:29.740325  7077 consensus_queue.cc:260] T 0cecd1b14e474dee825fae040ce1b31e P 94ffd7aaf85b4e17881241b8396fa2f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:29.740603  7076 ts_tablet_manager.cc:1398] T 086509f1ee0a4a81b2f8194b50dbe159 P 218bc8c1f9ce46059898fc738179adf9: Time spent starting tablet: real 0.029s	user 0.014s	sys 0.000s
I20240502 01:55:29.740563  7078 raft_consensus.cc:348] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 01:55:29.740833  7077 ts_tablet_manager.cc:1398] T 0cecd1b14e474dee825fae040ce1b31e P 94ffd7aaf85b4e17881241b8396fa2f3: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240502 01:55:29.741127  7076 tablet_bootstrap.cc:492] T 9a5f486852554e2fa2d07f7b9d686ced P 218bc8c1f9ce46059898fc738179adf9: Bootstrap starting.
I20240502 01:55:29.741381  7078 raft_consensus.cc:374] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:29.741557  7078 raft_consensus.cc:729] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c90ee440b44b708bccc2e2d5385cc0, State: Initialized, Role: FOLLOWER
I20240502 01:55:29.742257  7078 consensus_queue.cc:260] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 01:55:29.743649  7078 ts_tablet_manager.cc:1398] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 01:55:29.746268  7076 tablet_bootstrap.cc:654] T 9a5f486852554e2fa2d07f7b9d686ced P 218bc8c1f9ce46059898fc738179adf9: Neither blocks nor log segments found. Creating new log.
W20240502 01:55:29.747303  6383 tablet.cc:2369] T 0cecd1b14e474dee825fae040ce1b31e P 94ffd7aaf85b4e17881241b8396fa2f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:55:29.748440  7076 tablet_bootstrap.cc:492] T 9a5f486852554e2fa2d07f7b9d686ced P 218bc8c1f9ce46059898fc738179adf9: No bootstrap required, opened a new log
I20240502 01:55:29.748628  7076 ts_tablet_manager.cc:1367] T 9a5f486852554e2fa2d07f7b9d686ced P 218bc8c1f9ce46059898fc738179adf9: Time spent bootstrapping tablet: real 0.008s	user 0.004s	sys 0.000s
I20240502 01:55:29.749687  7076 raft_consensus.cc:348] T 9a5f486852554e2fa2d07f7b9d686ced P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:29.749884  7076 raft_consensus.cc:374] T 9a5f486852554e2fa2d07f7b9d686ced P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:29.750021  7076 raft_consensus.cc:729] T 9a5f486852554e2fa2d07f7b9d686ced P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218bc8c1f9ce46059898fc738179adf9, State: Initialized, Role: FOLLOWER
I20240502 01:55:29.750267  7076 consensus_queue.cc:260] T 9a5f486852554e2fa2d07f7b9d686ced P 218bc8c1f9ce46059898fc738179adf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:29.750752  7076 ts_tablet_manager.cc:1398] T 9a5f486852554e2fa2d07f7b9d686ced P 218bc8c1f9ce46059898fc738179adf9: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:55:29.751029  7076 tablet_bootstrap.cc:492] T 0cecd1b14e474dee825fae040ce1b31e P 218bc8c1f9ce46059898fc738179adf9: Bootstrap starting.
I20240502 01:55:29.752537  7076 tablet_bootstrap.cc:654] T 0cecd1b14e474dee825fae040ce1b31e P 218bc8c1f9ce46059898fc738179adf9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:29.754498  7076 tablet_bootstrap.cc:492] T 0cecd1b14e474dee825fae040ce1b31e P 218bc8c1f9ce46059898fc738179adf9: No bootstrap required, opened a new log
I20240502 01:55:29.754689  7076 ts_tablet_manager.cc:1367] T 0cecd1b14e474dee825fae040ce1b31e P 218bc8c1f9ce46059898fc738179adf9: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.003s
I20240502 01:55:29.755342  7076 raft_consensus.cc:348] T 0cecd1b14e474dee825fae040ce1b31e P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:29.755568  7076 raft_consensus.cc:374] T 0cecd1b14e474dee825fae040ce1b31e P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:29.755638  7076 raft_consensus.cc:729] T 0cecd1b14e474dee825fae040ce1b31e P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218bc8c1f9ce46059898fc738179adf9, State: Initialized, Role: FOLLOWER
I20240502 01:55:29.755928  7076 consensus_queue.cc:260] T 0cecd1b14e474dee825fae040ce1b31e P 218bc8c1f9ce46059898fc738179adf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:29.756510  7076 ts_tablet_manager.cc:1398] T 0cecd1b14e474dee825fae040ce1b31e P 218bc8c1f9ce46059898fc738179adf9: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 01:55:29.756814  7084 raft_consensus.cc:482] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:29.757094  7084 raft_consensus.cc:504] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:29.758996  7084 leader_election.cc:288] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227), 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245)
I20240502 01:55:29.766501  6068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cecd1b14e474dee825fae040ce1b31e" candidate_uuid: "07cdb213c77c4ffda1f999b66b725399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218bc8c1f9ce46059898fc738179adf9" is_pre_election: true
I20240502 01:55:29.766858  6336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cecd1b14e474dee825fae040ce1b31e" candidate_uuid: "07cdb213c77c4ffda1f999b66b725399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" is_pre_election: true
I20240502 01:55:29.767107  6068 raft_consensus.cc:2454] T 0cecd1b14e474dee825fae040ce1b31e P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07cdb213c77c4ffda1f999b66b725399 in term 0.
I20240502 01:55:29.767442  6336 raft_consensus.cc:2454] T 0cecd1b14e474dee825fae040ce1b31e P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07cdb213c77c4ffda1f999b66b725399 in term 0.
I20240502 01:55:29.767836  6402 leader_election.cc:302] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07cdb213c77c4ffda1f999b66b725399, 218bc8c1f9ce46059898fc738179adf9; no voters: 
I20240502 01:55:29.768311  7084 raft_consensus.cc:2781] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:29.768477  7084 raft_consensus.cc:482] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:29.768584  7084 raft_consensus.cc:3037] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:29.770704  7084 raft_consensus.cc:504] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:29.771370  7084 leader_election.cc:288] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 [CANDIDATE]: Term 1 election: Requested vote from peers 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227), 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245)
I20240502 01:55:29.771991  6068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cecd1b14e474dee825fae040ce1b31e" candidate_uuid: "07cdb213c77c4ffda1f999b66b725399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218bc8c1f9ce46059898fc738179adf9"
I20240502 01:55:29.772049  6336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cecd1b14e474dee825fae040ce1b31e" candidate_uuid: "07cdb213c77c4ffda1f999b66b725399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ffd7aaf85b4e17881241b8396fa2f3"
I20240502 01:55:29.772214  6068 raft_consensus.cc:3037] T 0cecd1b14e474dee825fae040ce1b31e P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:29.772243  6336 raft_consensus.cc:3037] T 0cecd1b14e474dee825fae040ce1b31e P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:29.774403  6336 raft_consensus.cc:2454] T 0cecd1b14e474dee825fae040ce1b31e P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07cdb213c77c4ffda1f999b66b725399 in term 1.
I20240502 01:55:29.774405  6068 raft_consensus.cc:2454] T 0cecd1b14e474dee825fae040ce1b31e P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07cdb213c77c4ffda1f999b66b725399 in term 1.
I20240502 01:55:29.774878  6402 leader_election.cc:302] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07cdb213c77c4ffda1f999b66b725399, 218bc8c1f9ce46059898fc738179adf9; no voters: 
I20240502 01:55:29.775226  7084 raft_consensus.cc:2781] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:29.775660  7084 raft_consensus.cc:686] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Becoming Leader. State: Replica: 07cdb213c77c4ffda1f999b66b725399, State: Running, Role: LEADER
I20240502 01:55:29.776238  7084 consensus_queue.cc:237] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
W20240502 01:55:29.779227  6251 tablet.cc:2369] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:55:29.780964  5920 catalog_manager.cc:5484] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132). New cstate: current_term: 1 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } }
W20240502 01:55:29.803153  6116 tablet.cc:2369] T 086509f1ee0a4a81b2f8194b50dbe159 P 218bc8c1f9ce46059898fc738179adf9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:55:29.832791  7086 raft_consensus.cc:482] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:29.833021  7086 raft_consensus.cc:504] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:29.834844  7086 leader_election.cc:288] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227), a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197)
I20240502 01:55:29.841831  6068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a5f486852554e2fa2d07f7b9d686ced" candidate_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218bc8c1f9ce46059898fc738179adf9" is_pre_election: true
I20240502 01:55:29.842088  6068 raft_consensus.cc:2454] T 9a5f486852554e2fa2d07f7b9d686ced P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94ffd7aaf85b4e17881241b8396fa2f3 in term 0.
I20240502 01:55:29.842244  6204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a5f486852554e2fa2d07f7b9d686ced" candidate_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" is_pre_election: true
I20240502 01:55:29.842592  6270 leader_election.cc:302] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9, 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 
I20240502 01:55:29.842825  6204 raft_consensus.cc:2454] T 9a5f486852554e2fa2d07f7b9d686ced P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94ffd7aaf85b4e17881241b8396fa2f3 in term 0.
I20240502 01:55:29.843096  7086 raft_consensus.cc:2781] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:29.843219  7086 raft_consensus.cc:482] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:29.843331  7086 raft_consensus.cc:3037] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:29.845621  7086 raft_consensus.cc:504] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:29.846283  7086 leader_election.cc:288] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 1 election: Requested vote from peers 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227), a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197)
I20240502 01:55:29.846720  6068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a5f486852554e2fa2d07f7b9d686ced" candidate_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218bc8c1f9ce46059898fc738179adf9"
I20240502 01:55:29.846796  6204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a5f486852554e2fa2d07f7b9d686ced" candidate_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c90ee440b44b708bccc2e2d5385cc0"
I20240502 01:55:29.846940  6068 raft_consensus.cc:3037] T 9a5f486852554e2fa2d07f7b9d686ced P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:29.846995  6204 raft_consensus.cc:3037] T 9a5f486852554e2fa2d07f7b9d686ced P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:29.848896  6068 raft_consensus.cc:2454] T 9a5f486852554e2fa2d07f7b9d686ced P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ffd7aaf85b4e17881241b8396fa2f3 in term 1.
I20240502 01:55:29.849146  6204 raft_consensus.cc:2454] T 9a5f486852554e2fa2d07f7b9d686ced P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ffd7aaf85b4e17881241b8396fa2f3 in term 1.
I20240502 01:55:29.849329  6270 leader_election.cc:302] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9, 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 
I20240502 01:55:29.849660  7086 raft_consensus.cc:2781] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:29.850111  7086 raft_consensus.cc:686] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Becoming Leader. State: Replica: 94ffd7aaf85b4e17881241b8396fa2f3, State: Running, Role: LEADER
I20240502 01:55:29.850661  7086 consensus_queue.cc:237] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:29.855276  5920 catalog_manager.cc:5484] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131). New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } }
I20240502 01:55:29.877281  7085 raft_consensus.cc:482] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:29.877521  7085 raft_consensus.cc:504] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 01:55:29.879281  7085 leader_election.cc:288] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867), 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227)
I20240502 01:55:29.886811  6068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086509f1ee0a4a81b2f8194b50dbe159" candidate_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218bc8c1f9ce46059898fc738179adf9" is_pre_election: true
I20240502 01:55:29.886888  6469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086509f1ee0a4a81b2f8194b50dbe159" candidate_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07cdb213c77c4ffda1f999b66b725399" is_pre_election: true
I20240502 01:55:29.887092  6068 raft_consensus.cc:2454] T 086509f1ee0a4a81b2f8194b50dbe159 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c90ee440b44b708bccc2e2d5385cc0 in term 0.
I20240502 01:55:29.887310  6469 raft_consensus.cc:2454] T 086509f1ee0a4a81b2f8194b50dbe159 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c90ee440b44b708bccc2e2d5385cc0 in term 0.
I20240502 01:55:29.887684  6136 leader_election.cc:302] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9, a2c90ee440b44b708bccc2e2d5385cc0; no voters: 
I20240502 01:55:29.888237  7085 raft_consensus.cc:2781] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:29.888378  7085 raft_consensus.cc:482] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:29.888495  7085 raft_consensus.cc:3037] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:29.890579  7085 raft_consensus.cc:504] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 01:55:29.891256  7085 leader_election.cc:288] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 [CANDIDATE]: Term 1 election: Requested vote from peers 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867), 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227)
I20240502 01:55:29.891942  6469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086509f1ee0a4a81b2f8194b50dbe159" candidate_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07cdb213c77c4ffda1f999b66b725399"
I20240502 01:55:29.891949  6068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086509f1ee0a4a81b2f8194b50dbe159" candidate_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218bc8c1f9ce46059898fc738179adf9"
I20240502 01:55:29.892153  6469 raft_consensus.cc:3037] T 086509f1ee0a4a81b2f8194b50dbe159 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:29.892150  6068 raft_consensus.cc:3037] T 086509f1ee0a4a81b2f8194b50dbe159 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:29.894143  6068 raft_consensus.cc:2454] T 086509f1ee0a4a81b2f8194b50dbe159 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c90ee440b44b708bccc2e2d5385cc0 in term 1.
I20240502 01:55:29.894222  6469 raft_consensus.cc:2454] T 086509f1ee0a4a81b2f8194b50dbe159 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c90ee440b44b708bccc2e2d5385cc0 in term 1.
I20240502 01:55:29.894701  6136 leader_election.cc:302] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9, a2c90ee440b44b708bccc2e2d5385cc0; no voters: 
I20240502 01:55:29.895069  7085 raft_consensus.cc:2781] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:29.895493  7085 raft_consensus.cc:686] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 LEADER]: Becoming Leader. State: Replica: a2c90ee440b44b708bccc2e2d5385cc0, State: Running, Role: LEADER
I20240502 01:55:29.896093  7085 consensus_queue.cc:237] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 01:55:29.900807  5920 catalog_manager.cc:5484] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130). New cstate: current_term: 1 leader_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } }
W20240502 01:55:29.970762  6515 tablet.cc:2369] T 086509f1ee0a4a81b2f8194b50dbe159 P 07cdb213c77c4ffda1f999b66b725399: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:55:30.188760  7084 consensus_queue.cc:1035] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:30.198701  7084 consensus_queue.cc:1035] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:55:30.295099  7106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:30.295477  7106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:30.303454  7106 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240502 01:55:30.340584  7086 consensus_queue.cc:1035] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:30.346868  7086 consensus_queue.cc:1035] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:30.448239  7085 consensus_queue.cc:1035] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:30.496379  6469 raft_consensus.cc:1261] T 086509f1ee0a4a81b2f8194b50dbe159 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Refusing update from remote peer a2c90ee440b44b708bccc2e2d5385cc0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:55:30.497511  7155 consensus_queue.cc:1035] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:30.517141  7151 mvcc.cc:205] Tried to move back new op lower bound from 7023062755229511680 to 7023062752858750976. Current Snapshot: MvccSnapshot[applied={T|T < 7023062755229511680}]

INSERT report
    rows total: 2000
    time total: 244.919 ms
  time per row: 0.12246 ms
I20240502 01:55:30.703716  5919 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58156:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:55:30.713239  6447 tablet_service.cc:1467] Processing CreateTablet for tablet 4eb91f5df851459b8ecf09cbe34fefa5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=0053d72412224a79be83b7e007b51934]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:30.713554  6182 tablet_service.cc:1467] Processing CreateTablet for tablet 4eb91f5df851459b8ecf09cbe34fefa5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=0053d72412224a79be83b7e007b51934]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:30.713804  6447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4eb91f5df851459b8ecf09cbe34fefa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:30.714011  6182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4eb91f5df851459b8ecf09cbe34fefa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:30.714133  6315 tablet_service.cc:1467] Processing CreateTablet for tablet 4eb91f5df851459b8ecf09cbe34fefa5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=0053d72412224a79be83b7e007b51934]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:30.714591  6315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4eb91f5df851459b8ecf09cbe34fefa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:30.714144  6316 tablet_service.cc:1467] Processing CreateTablet for tablet ac3325a06d8841718158e1f5f181a398 (DEFAULT_TABLE table=rebalance_test_table_1 [id=0053d72412224a79be83b7e007b51934]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:30.715088  6316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac3325a06d8841718158e1f5f181a398. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:30.717579  6448 tablet_service.cc:1467] Processing CreateTablet for tablet ac3325a06d8841718158e1f5f181a398 (DEFAULT_TABLE table=rebalance_test_table_1 [id=0053d72412224a79be83b7e007b51934]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:30.717978  6448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac3325a06d8841718158e1f5f181a398. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:30.719205  6181 tablet_service.cc:1467] Processing CreateTablet for tablet ac3325a06d8841718158e1f5f181a398 (DEFAULT_TABLE table=rebalance_test_table_1 [id=0053d72412224a79be83b7e007b51934]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:30.719605  6181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac3325a06d8841718158e1f5f181a398. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:30.721251  6180 tablet_service.cc:1467] Processing CreateTablet for tablet 89f7cf12d85442749e7d5b259ebca43f (DEFAULT_TABLE table=rebalance_test_table_1 [id=0053d72412224a79be83b7e007b51934]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:30.721823  6180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89f7cf12d85442749e7d5b259ebca43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:30.722584  6314 tablet_service.cc:1467] Processing CreateTablet for tablet 89f7cf12d85442749e7d5b259ebca43f (DEFAULT_TABLE table=rebalance_test_table_1 [id=0053d72412224a79be83b7e007b51934]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:30.723104  6314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89f7cf12d85442749e7d5b259ebca43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:30.724632  6048 tablet_service.cc:1467] Processing CreateTablet for tablet 89f7cf12d85442749e7d5b259ebca43f (DEFAULT_TABLE table=rebalance_test_table_1 [id=0053d72412224a79be83b7e007b51934]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:30.725083  6048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89f7cf12d85442749e7d5b259ebca43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:30.734319  7178 tablet_bootstrap.cc:492] T ac3325a06d8841718158e1f5f181a398 P a2c90ee440b44b708bccc2e2d5385cc0: Bootstrap starting.
I20240502 01:55:30.737257  7178 tablet_bootstrap.cc:654] T ac3325a06d8841718158e1f5f181a398 P a2c90ee440b44b708bccc2e2d5385cc0: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:30.738215  7177 tablet_bootstrap.cc:492] T 89f7cf12d85442749e7d5b259ebca43f P 218bc8c1f9ce46059898fc738179adf9: Bootstrap starting.
I20240502 01:55:30.740770  7177 tablet_bootstrap.cc:654] T 89f7cf12d85442749e7d5b259ebca43f P 218bc8c1f9ce46059898fc738179adf9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:30.746747  7180 tablet_bootstrap.cc:492] T ac3325a06d8841718158e1f5f181a398 P 07cdb213c77c4ffda1f999b66b725399: Bootstrap starting.
I20240502 01:55:30.749352  7177 tablet_bootstrap.cc:492] T 89f7cf12d85442749e7d5b259ebca43f P 218bc8c1f9ce46059898fc738179adf9: No bootstrap required, opened a new log
I20240502 01:55:30.749625  7177 ts_tablet_manager.cc:1367] T 89f7cf12d85442749e7d5b259ebca43f P 218bc8c1f9ce46059898fc738179adf9: Time spent bootstrapping tablet: real 0.012s	user 0.000s	sys 0.010s
I20240502 01:55:30.750562  7177 raft_consensus.cc:348] T 89f7cf12d85442749e7d5b259ebca43f P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:30.750950  7177 raft_consensus.cc:374] T 89f7cf12d85442749e7d5b259ebca43f P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:30.751034  7180 tablet_bootstrap.cc:654] T ac3325a06d8841718158e1f5f181a398 P 07cdb213c77c4ffda1f999b66b725399: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:30.751106  7177 raft_consensus.cc:729] T 89f7cf12d85442749e7d5b259ebca43f P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218bc8c1f9ce46059898fc738179adf9, State: Initialized, Role: FOLLOWER
I20240502 01:55:30.751556  7177 consensus_queue.cc:260] T 89f7cf12d85442749e7d5b259ebca43f P 218bc8c1f9ce46059898fc738179adf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:30.752691  7176 tablet_bootstrap.cc:492] T 4eb91f5df851459b8ecf09cbe34fefa5 P 94ffd7aaf85b4e17881241b8396fa2f3: Bootstrap starting.
I20240502 01:55:30.753013  7177 ts_tablet_manager.cc:1398] T 89f7cf12d85442749e7d5b259ebca43f P 218bc8c1f9ce46059898fc738179adf9: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:55:30.754699  7176 tablet_bootstrap.cc:654] T 4eb91f5df851459b8ecf09cbe34fefa5 P 94ffd7aaf85b4e17881241b8396fa2f3: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:30.757051  7178 tablet_bootstrap.cc:492] T ac3325a06d8841718158e1f5f181a398 P a2c90ee440b44b708bccc2e2d5385cc0: No bootstrap required, opened a new log
I20240502 01:55:30.757208  7180 tablet_bootstrap.cc:492] T ac3325a06d8841718158e1f5f181a398 P 07cdb213c77c4ffda1f999b66b725399: No bootstrap required, opened a new log
I20240502 01:55:30.757247  7178 ts_tablet_manager.cc:1367] T ac3325a06d8841718158e1f5f181a398 P a2c90ee440b44b708bccc2e2d5385cc0: Time spent bootstrapping tablet: real 0.023s	user 0.004s	sys 0.017s
I20240502 01:55:30.757375  7180 ts_tablet_manager.cc:1367] T ac3325a06d8841718158e1f5f181a398 P 07cdb213c77c4ffda1f999b66b725399: Time spent bootstrapping tablet: real 0.011s	user 0.003s	sys 0.000s
I20240502 01:55:30.758256  7176 tablet_bootstrap.cc:492] T 4eb91f5df851459b8ecf09cbe34fefa5 P 94ffd7aaf85b4e17881241b8396fa2f3: No bootstrap required, opened a new log
I20240502 01:55:30.758190  7178 raft_consensus.cc:348] T ac3325a06d8841718158e1f5f181a398 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:30.758222  7180 raft_consensus.cc:348] T ac3325a06d8841718158e1f5f181a398 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:30.758400  7176 ts_tablet_manager.cc:1367] T 4eb91f5df851459b8ecf09cbe34fefa5 P 94ffd7aaf85b4e17881241b8396fa2f3: Time spent bootstrapping tablet: real 0.006s	user 0.003s	sys 0.000s
I20240502 01:55:30.758426  7180 raft_consensus.cc:374] T ac3325a06d8841718158e1f5f181a398 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:30.758426  7178 raft_consensus.cc:374] T ac3325a06d8841718158e1f5f181a398 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:30.758513  7180 raft_consensus.cc:729] T ac3325a06d8841718158e1f5f181a398 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07cdb213c77c4ffda1f999b66b725399, State: Initialized, Role: FOLLOWER
I20240502 01:55:30.758513  7178 raft_consensus.cc:729] T ac3325a06d8841718158e1f5f181a398 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c90ee440b44b708bccc2e2d5385cc0, State: Initialized, Role: FOLLOWER
I20240502 01:55:30.758795  7180 consensus_queue.cc:260] T ac3325a06d8841718158e1f5f181a398 P 07cdb213c77c4ffda1f999b66b725399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:30.758786  7178 consensus_queue.cc:260] T ac3325a06d8841718158e1f5f181a398 P a2c90ee440b44b708bccc2e2d5385cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:30.759085  7176 raft_consensus.cc:348] T 4eb91f5df851459b8ecf09cbe34fefa5 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:30.759272  7176 raft_consensus.cc:374] T 4eb91f5df851459b8ecf09cbe34fefa5 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:30.759337  7176 raft_consensus.cc:729] T 4eb91f5df851459b8ecf09cbe34fefa5 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94ffd7aaf85b4e17881241b8396fa2f3, State: Initialized, Role: FOLLOWER
I20240502 01:55:30.759366  7178 ts_tablet_manager.cc:1398] T ac3325a06d8841718158e1f5f181a398 P a2c90ee440b44b708bccc2e2d5385cc0: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:55:30.759557  7176 consensus_queue.cc:260] T 4eb91f5df851459b8ecf09cbe34fefa5 P 94ffd7aaf85b4e17881241b8396fa2f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:30.760003  7180 ts_tablet_manager.cc:1398] T ac3325a06d8841718158e1f5f181a398 P 07cdb213c77c4ffda1f999b66b725399: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 01:55:30.760181  7176 ts_tablet_manager.cc:1398] T 4eb91f5df851459b8ecf09cbe34fefa5 P 94ffd7aaf85b4e17881241b8396fa2f3: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:55:30.760380  7180 tablet_bootstrap.cc:492] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399: Bootstrap starting.
I20240502 01:55:30.761497  7178 tablet_bootstrap.cc:492] T 4eb91f5df851459b8ecf09cbe34fefa5 P a2c90ee440b44b708bccc2e2d5385cc0: Bootstrap starting.
I20240502 01:55:30.761709  7176 tablet_bootstrap.cc:492] T ac3325a06d8841718158e1f5f181a398 P 94ffd7aaf85b4e17881241b8396fa2f3: Bootstrap starting.
I20240502 01:55:30.763438  7178 tablet_bootstrap.cc:654] T 4eb91f5df851459b8ecf09cbe34fefa5 P a2c90ee440b44b708bccc2e2d5385cc0: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:30.763726  7180 tablet_bootstrap.cc:654] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:30.764185  7176 tablet_bootstrap.cc:654] T ac3325a06d8841718158e1f5f181a398 P 94ffd7aaf85b4e17881241b8396fa2f3: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:30.767539  7176 tablet_bootstrap.cc:492] T ac3325a06d8841718158e1f5f181a398 P 94ffd7aaf85b4e17881241b8396fa2f3: No bootstrap required, opened a new log
I20240502 01:55:30.767728  7176 ts_tablet_manager.cc:1367] T ac3325a06d8841718158e1f5f181a398 P 94ffd7aaf85b4e17881241b8396fa2f3: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.000s
I20240502 01:55:30.768646  7178 tablet_bootstrap.cc:492] T 4eb91f5df851459b8ecf09cbe34fefa5 P a2c90ee440b44b708bccc2e2d5385cc0: No bootstrap required, opened a new log
I20240502 01:55:30.768625  7176 raft_consensus.cc:348] T ac3325a06d8841718158e1f5f181a398 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:30.768784  7178 ts_tablet_manager.cc:1367] T 4eb91f5df851459b8ecf09cbe34fefa5 P a2c90ee440b44b708bccc2e2d5385cc0: Time spent bootstrapping tablet: real 0.007s	user 0.002s	sys 0.001s
I20240502 01:55:30.768862  7176 raft_consensus.cc:374] T ac3325a06d8841718158e1f5f181a398 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:30.768926  7176 raft_consensus.cc:729] T ac3325a06d8841718158e1f5f181a398 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94ffd7aaf85b4e17881241b8396fa2f3, State: Initialized, Role: FOLLOWER
I20240502 01:55:30.769201  7176 consensus_queue.cc:260] T ac3325a06d8841718158e1f5f181a398 P 94ffd7aaf85b4e17881241b8396fa2f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:30.769381  7178 raft_consensus.cc:348] T 4eb91f5df851459b8ecf09cbe34fefa5 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:30.769587  7178 raft_consensus.cc:374] T 4eb91f5df851459b8ecf09cbe34fefa5 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:30.769647  7178 raft_consensus.cc:729] T 4eb91f5df851459b8ecf09cbe34fefa5 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c90ee440b44b708bccc2e2d5385cc0, State: Initialized, Role: FOLLOWER
I20240502 01:55:30.769754  7176 ts_tablet_manager.cc:1398] T ac3325a06d8841718158e1f5f181a398 P 94ffd7aaf85b4e17881241b8396fa2f3: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 01:55:30.769856  7178 consensus_queue.cc:260] T 4eb91f5df851459b8ecf09cbe34fefa5 P a2c90ee440b44b708bccc2e2d5385cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:30.770047  7176 tablet_bootstrap.cc:492] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3: Bootstrap starting.
I20240502 01:55:30.771096  7178 ts_tablet_manager.cc:1398] T 4eb91f5df851459b8ecf09cbe34fefa5 P a2c90ee440b44b708bccc2e2d5385cc0: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:55:30.771399  7178 tablet_bootstrap.cc:492] T 89f7cf12d85442749e7d5b259ebca43f P a2c90ee440b44b708bccc2e2d5385cc0: Bootstrap starting.
I20240502 01:55:30.774111  7178 tablet_bootstrap.cc:654] T 89f7cf12d85442749e7d5b259ebca43f P a2c90ee440b44b708bccc2e2d5385cc0: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:30.774333  7176 tablet_bootstrap.cc:654] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:30.776765  7180 tablet_bootstrap.cc:492] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399: No bootstrap required, opened a new log
I20240502 01:55:30.776975  7180 ts_tablet_manager.cc:1367] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399: Time spent bootstrapping tablet: real 0.017s	user 0.005s	sys 0.000s
I20240502 01:55:30.777020  7176 tablet_bootstrap.cc:492] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3: No bootstrap required, opened a new log
I20240502 01:55:30.777150  7176 ts_tablet_manager.cc:1367] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3: Time spent bootstrapping tablet: real 0.007s	user 0.003s	sys 0.000s
I20240502 01:55:30.777844  7180 raft_consensus.cc:348] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:30.777882  7176 raft_consensus.cc:348] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:30.778075  7176 raft_consensus.cc:374] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:30.778092  7180 raft_consensus.cc:374] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:30.778159  7180 raft_consensus.cc:729] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07cdb213c77c4ffda1f999b66b725399, State: Initialized, Role: FOLLOWER
I20240502 01:55:30.778167  7176 raft_consensus.cc:729] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94ffd7aaf85b4e17881241b8396fa2f3, State: Initialized, Role: FOLLOWER
I20240502 01:55:30.778391  7176 consensus_queue.cc:260] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:30.778415  7180 consensus_queue.cc:260] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:30.778923  7180 ts_tablet_manager.cc:1398] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 01:55:30.779527  7176 ts_tablet_manager.cc:1398] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:55:30.780400  7178 tablet_bootstrap.cc:492] T 89f7cf12d85442749e7d5b259ebca43f P a2c90ee440b44b708bccc2e2d5385cc0: No bootstrap required, opened a new log
I20240502 01:55:30.780551  7178 ts_tablet_manager.cc:1367] T 89f7cf12d85442749e7d5b259ebca43f P a2c90ee440b44b708bccc2e2d5385cc0: Time spent bootstrapping tablet: real 0.009s	user 0.003s	sys 0.000s
I20240502 01:55:30.781361  7178 raft_consensus.cc:348] T 89f7cf12d85442749e7d5b259ebca43f P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:30.781528  7178 raft_consensus.cc:374] T 89f7cf12d85442749e7d5b259ebca43f P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:30.781607  7178 raft_consensus.cc:729] T 89f7cf12d85442749e7d5b259ebca43f P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c90ee440b44b708bccc2e2d5385cc0, State: Initialized, Role: FOLLOWER
I20240502 01:55:30.781854  7178 consensus_queue.cc:260] T 89f7cf12d85442749e7d5b259ebca43f P a2c90ee440b44b708bccc2e2d5385cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:30.782265  7178 ts_tablet_manager.cc:1398] T 89f7cf12d85442749e7d5b259ebca43f P a2c90ee440b44b708bccc2e2d5385cc0: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 01:55:30.833837  7096 raft_consensus.cc:482] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:30.834045  7096 raft_consensus.cc:504] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:30.834932  7096 leader_election.cc:288] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197), 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245)
I20240502 01:55:30.835565  6336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eb91f5df851459b8ecf09cbe34fefa5" candidate_uuid: "07cdb213c77c4ffda1f999b66b725399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" is_pre_election: true
I20240502 01:55:30.835816  6336 raft_consensus.cc:2454] T 4eb91f5df851459b8ecf09cbe34fefa5 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07cdb213c77c4ffda1f999b66b725399 in term 0.
I20240502 01:55:30.836328  6402 leader_election.cc:302] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07cdb213c77c4ffda1f999b66b725399, 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 
I20240502 01:55:30.836663  7096 raft_consensus.cc:2781] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:30.836813  7096 raft_consensus.cc:482] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:30.836913  7096 raft_consensus.cc:3037] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:30.838992  7096 raft_consensus.cc:504] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:30.839668  7096 leader_election.cc:288] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 [CANDIDATE]: Term 1 election: Requested vote from peers a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197), 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245)
I20240502 01:55:30.840245  6336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eb91f5df851459b8ecf09cbe34fefa5" candidate_uuid: "07cdb213c77c4ffda1f999b66b725399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ffd7aaf85b4e17881241b8396fa2f3"
I20240502 01:55:30.840480  6336 raft_consensus.cc:3037] T 4eb91f5df851459b8ecf09cbe34fefa5 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:30.841715  6203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eb91f5df851459b8ecf09cbe34fefa5" candidate_uuid: "07cdb213c77c4ffda1f999b66b725399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" is_pre_election: true
I20240502 01:55:30.841836  6204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eb91f5df851459b8ecf09cbe34fefa5" candidate_uuid: "07cdb213c77c4ffda1f999b66b725399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c90ee440b44b708bccc2e2d5385cc0"
I20240502 01:55:30.841943  6203 raft_consensus.cc:2454] T 4eb91f5df851459b8ecf09cbe34fefa5 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07cdb213c77c4ffda1f999b66b725399 in term 0.
I20240502 01:55:30.842551  7086 raft_consensus.cc:482] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:30.842703  6336 raft_consensus.cc:2454] T 4eb91f5df851459b8ecf09cbe34fefa5 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07cdb213c77c4ffda1f999b66b725399 in term 1.
I20240502 01:55:30.842742  7086 raft_consensus.cc:504] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:30.843161  6402 leader_election.cc:302] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 07cdb213c77c4ffda1f999b66b725399, 94ffd7aaf85b4e17881241b8396fa2f3; no voters: a2c90ee440b44b708bccc2e2d5385cc0
I20240502 01:55:30.843475  7086 leader_election.cc:288] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197), 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227)
I20240502 01:55:30.843505  7096 raft_consensus.cc:2781] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:30.843734  7096 raft_consensus.cc:686] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Becoming Leader. State: Replica: 07cdb213c77c4ffda1f999b66b725399, State: Running, Role: LEADER
I20240502 01:55:30.844087  7096 consensus_queue.cc:237] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:30.844173  6067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89f7cf12d85442749e7d5b259ebca43f" candidate_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218bc8c1f9ce46059898fc738179adf9" is_pre_election: true
I20240502 01:55:30.844422  6067 raft_consensus.cc:2454] T 89f7cf12d85442749e7d5b259ebca43f P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94ffd7aaf85b4e17881241b8396fa2f3 in term 0.
I20240502 01:55:30.844950  6270 leader_election.cc:302] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9, 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 
I20240502 01:55:30.845050  6204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89f7cf12d85442749e7d5b259ebca43f" candidate_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" is_pre_election: true
I20240502 01:55:30.845220  7086 raft_consensus.cc:2781] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:30.845319  7086 raft_consensus.cc:482] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:30.845304  6204 raft_consensus.cc:2454] T 89f7cf12d85442749e7d5b259ebca43f P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94ffd7aaf85b4e17881241b8396fa2f3 in term 0.
I20240502 01:55:30.845424  7086 raft_consensus.cc:3037] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:30.847968  7086 raft_consensus.cc:504] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:30.847856  5919 catalog_manager.cc:5484] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132). New cstate: current_term: 1 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } }
I20240502 01:55:30.848716  7086 leader_election.cc:288] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 1 election: Requested vote from peers a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197), 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227)
I20240502 01:55:30.849131  6204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89f7cf12d85442749e7d5b259ebca43f" candidate_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c90ee440b44b708bccc2e2d5385cc0"
I20240502 01:55:30.849253  6067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89f7cf12d85442749e7d5b259ebca43f" candidate_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218bc8c1f9ce46059898fc738179adf9"
I20240502 01:55:30.849368  6204 raft_consensus.cc:3037] T 89f7cf12d85442749e7d5b259ebca43f P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:30.849476  6067 raft_consensus.cc:3037] T 89f7cf12d85442749e7d5b259ebca43f P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:30.851619  6204 raft_consensus.cc:2454] T 89f7cf12d85442749e7d5b259ebca43f P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ffd7aaf85b4e17881241b8396fa2f3 in term 1.
I20240502 01:55:30.852099  6271 leader_election.cc:302] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94ffd7aaf85b4e17881241b8396fa2f3, a2c90ee440b44b708bccc2e2d5385cc0; no voters: 
I20240502 01:55:30.852113  6067 raft_consensus.cc:2454] T 89f7cf12d85442749e7d5b259ebca43f P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ffd7aaf85b4e17881241b8396fa2f3 in term 1.
I20240502 01:55:30.852694  7086 raft_consensus.cc:2781] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:30.852856  7086 raft_consensus.cc:686] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Becoming Leader. State: Replica: 94ffd7aaf85b4e17881241b8396fa2f3, State: Running, Role: LEADER
I20240502 01:55:30.853093  7086 consensus_queue.cc:237] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:30.855702  5919 catalog_manager.cc:5484] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131). New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } }
I20240502 01:55:30.950040  7086 raft_consensus.cc:482] T ac3325a06d8841718158e1f5f181a398 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:30.950241  7086 raft_consensus.cc:504] T ac3325a06d8841718158e1f5f181a398 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:30.950971  7086 leader_election.cc:288] T ac3325a06d8841718158e1f5f181a398 P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197), 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867)
I20240502 01:55:30.951664  6204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3325a06d8841718158e1f5f181a398" candidate_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" is_pre_election: true
I20240502 01:55:30.951959  6204 raft_consensus.cc:2454] T ac3325a06d8841718158e1f5f181a398 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94ffd7aaf85b4e17881241b8396fa2f3 in term 0.
I20240502 01:55:30.952383  6271 leader_election.cc:302] T ac3325a06d8841718158e1f5f181a398 P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94ffd7aaf85b4e17881241b8396fa2f3, a2c90ee440b44b708bccc2e2d5385cc0; no voters: 
I20240502 01:55:30.952746  7086 raft_consensus.cc:2781] T ac3325a06d8841718158e1f5f181a398 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:30.952898  7086 raft_consensus.cc:482] T ac3325a06d8841718158e1f5f181a398 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:30.952988  7086 raft_consensus.cc:3037] T ac3325a06d8841718158e1f5f181a398 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:30.955485  7086 raft_consensus.cc:504] T ac3325a06d8841718158e1f5f181a398 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:30.956532  7086 leader_election.cc:288] T ac3325a06d8841718158e1f5f181a398 P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 1 election: Requested vote from peers a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197), 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867)
I20240502 01:55:30.957151  6204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3325a06d8841718158e1f5f181a398" candidate_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c90ee440b44b708bccc2e2d5385cc0"
I20240502 01:55:30.957397  6204 raft_consensus.cc:3037] T ac3325a06d8841718158e1f5f181a398 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:30.958001  6469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3325a06d8841718158e1f5f181a398" candidate_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07cdb213c77c4ffda1f999b66b725399" is_pre_election: true
I20240502 01:55:30.958273  6469 raft_consensus.cc:2454] T ac3325a06d8841718158e1f5f181a398 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94ffd7aaf85b4e17881241b8396fa2f3 in term 0.
I20240502 01:55:30.958145  6467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3325a06d8841718158e1f5f181a398" candidate_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07cdb213c77c4ffda1f999b66b725399"
I20240502 01:55:30.958446  6467 raft_consensus.cc:3037] T ac3325a06d8841718158e1f5f181a398 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:30.959690  6204 raft_consensus.cc:2454] T ac3325a06d8841718158e1f5f181a398 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ffd7aaf85b4e17881241b8396fa2f3 in term 1.
I20240502 01:55:30.960166  6271 leader_election.cc:302] T ac3325a06d8841718158e1f5f181a398 P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94ffd7aaf85b4e17881241b8396fa2f3, a2c90ee440b44b708bccc2e2d5385cc0; no voters: 
I20240502 01:55:30.960480  7086 raft_consensus.cc:2781] T ac3325a06d8841718158e1f5f181a398 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:30.960609  7086 raft_consensus.cc:686] T ac3325a06d8841718158e1f5f181a398 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Becoming Leader. State: Replica: 94ffd7aaf85b4e17881241b8396fa2f3, State: Running, Role: LEADER
I20240502 01:55:30.960628  6467 raft_consensus.cc:2454] T ac3325a06d8841718158e1f5f181a398 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ffd7aaf85b4e17881241b8396fa2f3 in term 1.
I20240502 01:55:30.960930  7086 consensus_queue.cc:237] T ac3325a06d8841718158e1f5f181a398 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:30.963482  5919 catalog_manager.cc:5484] T ac3325a06d8841718158e1f5f181a398 P 94ffd7aaf85b4e17881241b8396fa2f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131). New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } }
I20240502 01:55:31.255517  6204 raft_consensus.cc:3037] T 4eb91f5df851459b8ecf09cbe34fefa5 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:31.258710  7096 consensus_queue.cc:1035] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:31.264523  7096 consensus_queue.cc:1035] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:31.281111  7086 consensus_queue.cc:1035] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:31.286721  7100 consensus_queue.cc:1035] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:55:31.343173  7194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:31.343495  7194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:31.351219  7194 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:55:31.372717  7202 consensus_queue.cc:1035] T ac3325a06d8841718158e1f5f181a398 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:31.377276  7086 consensus_queue.cc:1035] T ac3325a06d8841718158e1f5f181a398 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'

INSERT report
    rows total: 2000
    time total: 257.411 ms
  time per row: 0.128706 ms
I20240502 01:55:31.762380  5921 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58156:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:55:31.772182  6448 tablet_service.cc:1467] Processing CreateTablet for tablet 02b106ec57b94e44a29947a80a030a23 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ef7ebedfbbff4054b074bf7a7622b632]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:31.772182  6048 tablet_service.cc:1467] Processing CreateTablet for tablet 02b106ec57b94e44a29947a80a030a23 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ef7ebedfbbff4054b074bf7a7622b632]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:31.772691  6048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02b106ec57b94e44a29947a80a030a23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:31.772691  6448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02b106ec57b94e44a29947a80a030a23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:31.772701  6180 tablet_service.cc:1467] Processing CreateTablet for tablet 02b106ec57b94e44a29947a80a030a23 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ef7ebedfbbff4054b074bf7a7622b632]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:31.773211  6180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02b106ec57b94e44a29947a80a030a23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:31.775089  6446 tablet_service.cc:1467] Processing CreateTablet for tablet ca458907378e42d48adaa0ee647db4cd (DEFAULT_TABLE table=rebalance_test_table_2 [id=ef7ebedfbbff4054b074bf7a7622b632]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:31.775606  6446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca458907378e42d48adaa0ee647db4cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:31.776691  6315 tablet_service.cc:1467] Processing CreateTablet for tablet afa5561f8c0e4c0fa9bbbd5da61348c7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ef7ebedfbbff4054b074bf7a7622b632]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:31.777158  6315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afa5561f8c0e4c0fa9bbbd5da61348c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:31.780488  6447 tablet_service.cc:1467] Processing CreateTablet for tablet afa5561f8c0e4c0fa9bbbd5da61348c7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ef7ebedfbbff4054b074bf7a7622b632]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:31.780974  6447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afa5561f8c0e4c0fa9bbbd5da61348c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:31.780979  6046 tablet_service.cc:1467] Processing CreateTablet for tablet afa5561f8c0e4c0fa9bbbd5da61348c7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ef7ebedfbbff4054b074bf7a7622b632]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:31.781410  6046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afa5561f8c0e4c0fa9bbbd5da61348c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:31.785409  6181 tablet_service.cc:1467] Processing CreateTablet for tablet ca458907378e42d48adaa0ee647db4cd (DEFAULT_TABLE table=rebalance_test_table_2 [id=ef7ebedfbbff4054b074bf7a7622b632]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:31.785903  6181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca458907378e42d48adaa0ee647db4cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:31.786859  6047 tablet_service.cc:1467] Processing CreateTablet for tablet ca458907378e42d48adaa0ee647db4cd (DEFAULT_TABLE table=rebalance_test_table_2 [id=ef7ebedfbbff4054b074bf7a7622b632]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:31.787274  6047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca458907378e42d48adaa0ee647db4cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:31.788408  7262 tablet_bootstrap.cc:492] T 02b106ec57b94e44a29947a80a030a23 P 218bc8c1f9ce46059898fc738179adf9: Bootstrap starting.
I20240502 01:55:31.790433  7262 tablet_bootstrap.cc:654] T 02b106ec57b94e44a29947a80a030a23 P 218bc8c1f9ce46059898fc738179adf9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:31.796332  7262 tablet_bootstrap.cc:492] T 02b106ec57b94e44a29947a80a030a23 P 218bc8c1f9ce46059898fc738179adf9: No bootstrap required, opened a new log
I20240502 01:55:31.796532  7262 ts_tablet_manager.cc:1367] T 02b106ec57b94e44a29947a80a030a23 P 218bc8c1f9ce46059898fc738179adf9: Time spent bootstrapping tablet: real 0.008s	user 0.000s	sys 0.003s
I20240502 01:55:31.797420  7262 raft_consensus.cc:348] T 02b106ec57b94e44a29947a80a030a23 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:31.798336  7260 tablet_bootstrap.cc:492] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399: Bootstrap starting.
I20240502 01:55:31.798382  7262 raft_consensus.cc:374] T 02b106ec57b94e44a29947a80a030a23 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:31.798480  7262 raft_consensus.cc:729] T 02b106ec57b94e44a29947a80a030a23 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218bc8c1f9ce46059898fc738179adf9, State: Initialized, Role: FOLLOWER
I20240502 01:55:31.798609  7263 tablet_bootstrap.cc:492] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3: Bootstrap starting.
I20240502 01:55:31.798801  7262 consensus_queue.cc:260] T 02b106ec57b94e44a29947a80a030a23 P 218bc8c1f9ce46059898fc738179adf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:31.801367  7260 tablet_bootstrap.cc:654] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:31.801932  7263 tablet_bootstrap.cc:654] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:31.806612  7263 tablet_bootstrap.cc:492] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3: No bootstrap required, opened a new log
I20240502 01:55:31.806805  7263 ts_tablet_manager.cc:1367] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3: Time spent bootstrapping tablet: real 0.008s	user 0.003s	sys 0.000s
I20240502 01:55:31.807696  7263 raft_consensus.cc:348] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:31.807996  7263 raft_consensus.cc:374] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:31.808094  7263 raft_consensus.cc:729] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94ffd7aaf85b4e17881241b8396fa2f3, State: Initialized, Role: FOLLOWER
I20240502 01:55:31.809551  7261 tablet_bootstrap.cc:492] T 02b106ec57b94e44a29947a80a030a23 P a2c90ee440b44b708bccc2e2d5385cc0: Bootstrap starting.
I20240502 01:55:31.810668  7260 tablet_bootstrap.cc:492] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399: No bootstrap required, opened a new log
I20240502 01:55:31.810863  7260 ts_tablet_manager.cc:1367] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399: Time spent bootstrapping tablet: real 0.013s	user 0.005s	sys 0.005s
I20240502 01:55:31.811914  7260 raft_consensus.cc:348] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:31.812139  7260 raft_consensus.cc:374] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:31.812237  7260 raft_consensus.cc:729] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07cdb213c77c4ffda1f999b66b725399, State: Initialized, Role: FOLLOWER
I20240502 01:55:31.812065  7263 consensus_queue.cc:260] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:31.812577  7262 ts_tablet_manager.cc:1398] T 02b106ec57b94e44a29947a80a030a23 P 218bc8c1f9ce46059898fc738179adf9: Time spent starting tablet: real 0.016s	user 0.013s	sys 0.003s
I20240502 01:55:31.812716  7260 consensus_queue.cc:260] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:31.812922  7262 tablet_bootstrap.cc:492] T ca458907378e42d48adaa0ee647db4cd P 218bc8c1f9ce46059898fc738179adf9: Bootstrap starting.
I20240502 01:55:31.813388  7263 ts_tablet_manager.cc:1398] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.000s
I20240502 01:55:31.814004  7261 tablet_bootstrap.cc:654] T 02b106ec57b94e44a29947a80a030a23 P a2c90ee440b44b708bccc2e2d5385cc0: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:31.814925  7260 ts_tablet_manager.cc:1398] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.003s
I20240502 01:55:31.815258  7260 tablet_bootstrap.cc:492] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 07cdb213c77c4ffda1f999b66b725399: Bootstrap starting.
I20240502 01:55:31.817061  7260 tablet_bootstrap.cc:654] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 07cdb213c77c4ffda1f999b66b725399: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:31.818475  7261 tablet_bootstrap.cc:492] T 02b106ec57b94e44a29947a80a030a23 P a2c90ee440b44b708bccc2e2d5385cc0: No bootstrap required, opened a new log
I20240502 01:55:31.818656  7261 ts_tablet_manager.cc:1367] T 02b106ec57b94e44a29947a80a030a23 P a2c90ee440b44b708bccc2e2d5385cc0: Time spent bootstrapping tablet: real 0.009s	user 0.005s	sys 0.000s
I20240502 01:55:31.819630  7262 tablet_bootstrap.cc:654] T ca458907378e42d48adaa0ee647db4cd P 218bc8c1f9ce46059898fc738179adf9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:31.819494  7261 raft_consensus.cc:348] T 02b106ec57b94e44a29947a80a030a23 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:31.819715  7261 raft_consensus.cc:374] T 02b106ec57b94e44a29947a80a030a23 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:31.819797  7261 raft_consensus.cc:729] T 02b106ec57b94e44a29947a80a030a23 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c90ee440b44b708bccc2e2d5385cc0, State: Initialized, Role: FOLLOWER
I20240502 01:55:31.820108  7261 consensus_queue.cc:260] T 02b106ec57b94e44a29947a80a030a23 P a2c90ee440b44b708bccc2e2d5385cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:31.820677  7261 ts_tablet_manager.cc:1398] T 02b106ec57b94e44a29947a80a030a23 P a2c90ee440b44b708bccc2e2d5385cc0: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:55:31.820983  7261 tablet_bootstrap.cc:492] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0: Bootstrap starting.
I20240502 01:55:31.824241  7261 tablet_bootstrap.cc:654] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:31.824627  7262 tablet_bootstrap.cc:492] T ca458907378e42d48adaa0ee647db4cd P 218bc8c1f9ce46059898fc738179adf9: No bootstrap required, opened a new log
I20240502 01:55:31.824823  7262 ts_tablet_manager.cc:1367] T ca458907378e42d48adaa0ee647db4cd P 218bc8c1f9ce46059898fc738179adf9: Time spent bootstrapping tablet: real 0.012s	user 0.003s	sys 0.002s
I20240502 01:55:31.824992  7260 tablet_bootstrap.cc:492] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 07cdb213c77c4ffda1f999b66b725399: No bootstrap required, opened a new log
I20240502 01:55:31.825170  7260 ts_tablet_manager.cc:1367] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 07cdb213c77c4ffda1f999b66b725399: Time spent bootstrapping tablet: real 0.010s	user 0.002s	sys 0.006s
I20240502 01:55:31.825690  7262 raft_consensus.cc:348] T ca458907378e42d48adaa0ee647db4cd P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:31.825928  7262 raft_consensus.cc:374] T ca458907378e42d48adaa0ee647db4cd P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:31.825978  7261 tablet_bootstrap.cc:492] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0: No bootstrap required, opened a new log
I20240502 01:55:31.826017  7262 raft_consensus.cc:729] T ca458907378e42d48adaa0ee647db4cd P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218bc8c1f9ce46059898fc738179adf9, State: Initialized, Role: FOLLOWER
I20240502 01:55:31.825955  7260 raft_consensus.cc:348] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:31.826140  7261 ts_tablet_manager.cc:1367] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0: Time spent bootstrapping tablet: real 0.005s	user 0.001s	sys 0.002s
I20240502 01:55:31.826184  7260 raft_consensus.cc:374] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:31.826260  7260 raft_consensus.cc:729] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07cdb213c77c4ffda1f999b66b725399, State: Initialized, Role: FOLLOWER
I20240502 01:55:31.826328  7262 consensus_queue.cc:260] T ca458907378e42d48adaa0ee647db4cd P 218bc8c1f9ce46059898fc738179adf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:31.826515  7260 consensus_queue.cc:260] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 07cdb213c77c4ffda1f999b66b725399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:31.826905  7262 ts_tablet_manager.cc:1398] T ca458907378e42d48adaa0ee647db4cd P 218bc8c1f9ce46059898fc738179adf9: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240502 01:55:31.827082  7261 raft_consensus.cc:348] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:31.827270  7262 tablet_bootstrap.cc:492] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9: Bootstrap starting.
I20240502 01:55:31.827288  7261 raft_consensus.cc:374] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:31.827371  7261 raft_consensus.cc:729] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c90ee440b44b708bccc2e2d5385cc0, State: Initialized, Role: FOLLOWER
I20240502 01:55:31.827641  7261 consensus_queue.cc:260] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:31.828348  7261 ts_tablet_manager.cc:1398] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240502 01:55:31.828969  7260 ts_tablet_manager.cc:1398] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 07cdb213c77c4ffda1f999b66b725399: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.002s
I20240502 01:55:31.829350  7260 tablet_bootstrap.cc:492] T ca458907378e42d48adaa0ee647db4cd P 07cdb213c77c4ffda1f999b66b725399: Bootstrap starting.
I20240502 01:55:31.829556  7262 tablet_bootstrap.cc:654] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:31.831038  7260 tablet_bootstrap.cc:654] T ca458907378e42d48adaa0ee647db4cd P 07cdb213c77c4ffda1f999b66b725399: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:31.833055  7260 tablet_bootstrap.cc:492] T ca458907378e42d48adaa0ee647db4cd P 07cdb213c77c4ffda1f999b66b725399: No bootstrap required, opened a new log
I20240502 01:55:31.833201  7260 ts_tablet_manager.cc:1367] T ca458907378e42d48adaa0ee647db4cd P 07cdb213c77c4ffda1f999b66b725399: Time spent bootstrapping tablet: real 0.004s	user 0.001s	sys 0.002s
I20240502 01:55:31.833719  7262 tablet_bootstrap.cc:492] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9: No bootstrap required, opened a new log
I20240502 01:55:31.833874  7262 ts_tablet_manager.cc:1367] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9: Time spent bootstrapping tablet: real 0.007s	user 0.003s	sys 0.002s
I20240502 01:55:31.833932  7260 raft_consensus.cc:348] T ca458907378e42d48adaa0ee647db4cd P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:31.834120  7260 raft_consensus.cc:374] T ca458907378e42d48adaa0ee647db4cd P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:31.834208  7260 raft_consensus.cc:729] T ca458907378e42d48adaa0ee647db4cd P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07cdb213c77c4ffda1f999b66b725399, State: Initialized, Role: FOLLOWER
I20240502 01:55:31.834450  7260 consensus_queue.cc:260] T ca458907378e42d48adaa0ee647db4cd P 07cdb213c77c4ffda1f999b66b725399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:31.834627  7262 raft_consensus.cc:348] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:31.834827  7262 raft_consensus.cc:374] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:31.834916  7262 raft_consensus.cc:729] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218bc8c1f9ce46059898fc738179adf9, State: Initialized, Role: FOLLOWER
I20240502 01:55:31.835084  7260 ts_tablet_manager.cc:1398] T ca458907378e42d48adaa0ee647db4cd P 07cdb213c77c4ffda1f999b66b725399: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 01:55:31.835215  7262 consensus_queue.cc:260] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:31.835765  7262 ts_tablet_manager.cc:1398] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240502 01:55:31.846678  7086 raft_consensus.cc:482] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:31.846928  7086 raft_consensus.cc:504] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:31.847586  7086 leader_election.cc:288] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227), 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867)
I20240502 01:55:31.848110  6067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afa5561f8c0e4c0fa9bbbd5da61348c7" candidate_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218bc8c1f9ce46059898fc738179adf9" is_pre_election: true
I20240502 01:55:31.848165  6467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afa5561f8c0e4c0fa9bbbd5da61348c7" candidate_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07cdb213c77c4ffda1f999b66b725399" is_pre_election: true
I20240502 01:55:31.848376  6067 raft_consensus.cc:2454] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94ffd7aaf85b4e17881241b8396fa2f3 in term 0.
I20240502 01:55:31.848407  6467 raft_consensus.cc:2454] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94ffd7aaf85b4e17881241b8396fa2f3 in term 0.
I20240502 01:55:31.848842  6270 leader_election.cc:302] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9, 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 
I20240502 01:55:31.849145  7086 raft_consensus.cc:2781] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:31.849229  7086 raft_consensus.cc:482] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:31.849303  7086 raft_consensus.cc:3037] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:31.851225  7086 raft_consensus.cc:504] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:31.851847  7086 leader_election.cc:288] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 1 election: Requested vote from peers 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227), 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867)
I20240502 01:55:31.852401  6067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afa5561f8c0e4c0fa9bbbd5da61348c7" candidate_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218bc8c1f9ce46059898fc738179adf9"
I20240502 01:55:31.852469  6467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afa5561f8c0e4c0fa9bbbd5da61348c7" candidate_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07cdb213c77c4ffda1f999b66b725399"
I20240502 01:55:31.852674  6067 raft_consensus.cc:3037] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:31.852684  6467 raft_consensus.cc:3037] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:31.855569  6067 raft_consensus.cc:2454] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ffd7aaf85b4e17881241b8396fa2f3 in term 1.
I20240502 01:55:31.855569  6467 raft_consensus.cc:2454] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ffd7aaf85b4e17881241b8396fa2f3 in term 1.
I20240502 01:55:31.856135  6268 leader_election.cc:302] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07cdb213c77c4ffda1f999b66b725399, 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 
I20240502 01:55:31.856526  7086 raft_consensus.cc:2781] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:31.856701  7086 raft_consensus.cc:686] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Becoming Leader. State: Replica: 94ffd7aaf85b4e17881241b8396fa2f3, State: Running, Role: LEADER
I20240502 01:55:31.856984  7086 consensus_queue.cc:237] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:31.859768  5921 catalog_manager.cc:5484] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131). New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } }
I20240502 01:55:32.016414  7084 raft_consensus.cc:482] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:32.016659  7084 raft_consensus.cc:504] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:32.017324  7084 leader_election.cc:288] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227), a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197)
I20240502 01:55:32.017938  6067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02b106ec57b94e44a29947a80a030a23" candidate_uuid: "07cdb213c77c4ffda1f999b66b725399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218bc8c1f9ce46059898fc738179adf9" is_pre_election: true
I20240502 01:55:32.017944  6203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02b106ec57b94e44a29947a80a030a23" candidate_uuid: "07cdb213c77c4ffda1f999b66b725399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" is_pre_election: true
I20240502 01:55:32.018154  6203 raft_consensus.cc:2454] T 02b106ec57b94e44a29947a80a030a23 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07cdb213c77c4ffda1f999b66b725399 in term 0.
I20240502 01:55:32.018173  6067 raft_consensus.cc:2454] T 02b106ec57b94e44a29947a80a030a23 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07cdb213c77c4ffda1f999b66b725399 in term 0.
I20240502 01:55:32.018620  6402 leader_election.cc:302] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07cdb213c77c4ffda1f999b66b725399, 218bc8c1f9ce46059898fc738179adf9; no voters: 
I20240502 01:55:32.018929  7084 raft_consensus.cc:2781] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:32.019052  7084 raft_consensus.cc:482] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:32.019176  7084 raft_consensus.cc:3037] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:32.021297  7084 raft_consensus.cc:504] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:32.021971  7084 leader_election.cc:288] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 [CANDIDATE]: Term 1 election: Requested vote from peers 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227), a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197)
I20240502 01:55:32.022451  6203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02b106ec57b94e44a29947a80a030a23" candidate_uuid: "07cdb213c77c4ffda1f999b66b725399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c90ee440b44b708bccc2e2d5385cc0"
I20240502 01:55:32.022655  6203 raft_consensus.cc:3037] T 02b106ec57b94e44a29947a80a030a23 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:32.022648  6067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02b106ec57b94e44a29947a80a030a23" candidate_uuid: "07cdb213c77c4ffda1f999b66b725399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218bc8c1f9ce46059898fc738179adf9"
I20240502 01:55:32.022859  6067 raft_consensus.cc:3037] T 02b106ec57b94e44a29947a80a030a23 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:32.024642  6203 raft_consensus.cc:2454] T 02b106ec57b94e44a29947a80a030a23 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07cdb213c77c4ffda1f999b66b725399 in term 1.
I20240502 01:55:32.024756  6067 raft_consensus.cc:2454] T 02b106ec57b94e44a29947a80a030a23 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07cdb213c77c4ffda1f999b66b725399 in term 1.
I20240502 01:55:32.025079  6403 leader_election.cc:302] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07cdb213c77c4ffda1f999b66b725399, a2c90ee440b44b708bccc2e2d5385cc0; no voters: 
I20240502 01:55:32.025391  7254 raft_consensus.cc:2781] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:32.025527  7254 raft_consensus.cc:686] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Becoming Leader. State: Replica: 07cdb213c77c4ffda1f999b66b725399, State: Running, Role: LEADER
I20240502 01:55:32.025805  7254 consensus_queue.cc:237] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:32.028563  5921 catalog_manager.cc:5484] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132). New cstate: current_term: 1 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } }
I20240502 01:55:32.122401  7150 raft_consensus.cc:482] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:32.122604  7150 raft_consensus.cc:504] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:32.123288  7150 leader_election.cc:288] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227), 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867)
I20240502 01:55:32.123929  6067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca458907378e42d48adaa0ee647db4cd" candidate_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218bc8c1f9ce46059898fc738179adf9" is_pre_election: true
I20240502 01:55:32.123935  6467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca458907378e42d48adaa0ee647db4cd" candidate_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07cdb213c77c4ffda1f999b66b725399" is_pre_election: true
I20240502 01:55:32.124166  6467 raft_consensus.cc:2454] T ca458907378e42d48adaa0ee647db4cd P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c90ee440b44b708bccc2e2d5385cc0 in term 0.
I20240502 01:55:32.124179  6067 raft_consensus.cc:2454] T ca458907378e42d48adaa0ee647db4cd P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c90ee440b44b708bccc2e2d5385cc0 in term 0.
I20240502 01:55:32.124634  6136 leader_election.cc:302] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9, a2c90ee440b44b708bccc2e2d5385cc0; no voters: 
I20240502 01:55:32.124922  7150 raft_consensus.cc:2781] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:32.125027  7150 raft_consensus.cc:482] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:32.125092  7150 raft_consensus.cc:3037] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:32.127048  7150 raft_consensus.cc:504] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:32.127667  7150 leader_election.cc:288] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 [CANDIDATE]: Term 1 election: Requested vote from peers 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227), 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867)
I20240502 01:55:32.128319  6067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca458907378e42d48adaa0ee647db4cd" candidate_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218bc8c1f9ce46059898fc738179adf9"
I20240502 01:55:32.128351  6467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca458907378e42d48adaa0ee647db4cd" candidate_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07cdb213c77c4ffda1f999b66b725399"
I20240502 01:55:32.128543  6067 raft_consensus.cc:3037] T ca458907378e42d48adaa0ee647db4cd P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:32.128551  6467 raft_consensus.cc:3037] T ca458907378e42d48adaa0ee647db4cd P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:32.130402  6067 raft_consensus.cc:2454] T ca458907378e42d48adaa0ee647db4cd P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c90ee440b44b708bccc2e2d5385cc0 in term 1.
I20240502 01:55:32.130614  6467 raft_consensus.cc:2454] T ca458907378e42d48adaa0ee647db4cd P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c90ee440b44b708bccc2e2d5385cc0 in term 1.
I20240502 01:55:32.130877  6136 leader_election.cc:302] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9, a2c90ee440b44b708bccc2e2d5385cc0; no voters: 
I20240502 01:55:32.131230  7150 raft_consensus.cc:2781] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:32.131601  7150 raft_consensus.cc:686] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 LEADER]: Becoming Leader. State: Replica: a2c90ee440b44b708bccc2e2d5385cc0, State: Running, Role: LEADER
I20240502 01:55:32.131953  7150 consensus_queue.cc:237] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:32.134770  5921 catalog_manager.cc:5484] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130). New cstate: current_term: 1 leader_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } }
I20240502 01:55:32.356626  7086 consensus_queue.cc:1035] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:32.364037  7086 consensus_queue.cc:1035] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:32.430107  7254 consensus_queue.cc:1035] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:32.434963  7289 consensus_queue.cc:1035] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:55:32.537597  7278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:32.537957  7278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:32.545544  7278 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 01:55:32.644750  7150 consensus_queue.cc:1035] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:32.676257  7317 consensus_queue.cc:1035] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 290.954 ms
  time per row: 0.145477 ms
I20240502 01:55:32.986372  5916 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58156:
name: "rebalance_test_table_3"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:55:32.998849  6181 tablet_service.cc:1467] Processing CreateTablet for tablet 7f2519b0c6404799ad84d120b45a7ea4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=63a163be603345428cc84c86b96467f0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:32.999398  6181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f2519b0c6404799ad84d120b45a7ea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:33.005095  6046 tablet_service.cc:1467] Processing CreateTablet for tablet 7f2519b0c6404799ad84d120b45a7ea4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=63a163be603345428cc84c86b96467f0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:33.005589  6046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f2519b0c6404799ad84d120b45a7ea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:33.005987  6314 tablet_service.cc:1467] Processing CreateTablet for tablet 6984d638991442e3adaed1b24bee95e9 (DEFAULT_TABLE table=rebalance_test_table_3 [id=63a163be603345428cc84c86b96467f0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:33.006042  6047 tablet_service.cc:1467] Processing CreateTablet for tablet 6984d638991442e3adaed1b24bee95e9 (DEFAULT_TABLE table=rebalance_test_table_3 [id=63a163be603345428cc84c86b96467f0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:33.006251  6315 tablet_service.cc:1467] Processing CreateTablet for tablet 7f2519b0c6404799ad84d120b45a7ea4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=63a163be603345428cc84c86b96467f0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:33.006455  6047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6984d638991442e3adaed1b24bee95e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:33.006503  6314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6984d638991442e3adaed1b24bee95e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:33.006736  6315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f2519b0c6404799ad84d120b45a7ea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:33.012455  6180 tablet_service.cc:1467] Processing CreateTablet for tablet 866f87ea9f7f48edb667b1b2eb3fa239 (DEFAULT_TABLE table=rebalance_test_table_3 [id=63a163be603345428cc84c86b96467f0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:33.013051  6180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 866f87ea9f7f48edb667b1b2eb3fa239. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:33.013317  6446 tablet_service.cc:1467] Processing CreateTablet for tablet 6984d638991442e3adaed1b24bee95e9 (DEFAULT_TABLE table=rebalance_test_table_3 [id=63a163be603345428cc84c86b96467f0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:33.013748  6446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6984d638991442e3adaed1b24bee95e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:33.016484  7345 tablet_bootstrap.cc:492] T 7f2519b0c6404799ad84d120b45a7ea4 P 218bc8c1f9ce46059898fc738179adf9: Bootstrap starting.
I20240502 01:55:33.017601  6447 tablet_service.cc:1467] Processing CreateTablet for tablet 866f87ea9f7f48edb667b1b2eb3fa239 (DEFAULT_TABLE table=rebalance_test_table_3 [id=63a163be603345428cc84c86b96467f0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:33.018080  6447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 866f87ea9f7f48edb667b1b2eb3fa239. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:33.021345  7346 tablet_bootstrap.cc:492] T 7f2519b0c6404799ad84d120b45a7ea4 P a2c90ee440b44b708bccc2e2d5385cc0: Bootstrap starting.
I20240502 01:55:33.021651  7345 tablet_bootstrap.cc:654] T 7f2519b0c6404799ad84d120b45a7ea4 P 218bc8c1f9ce46059898fc738179adf9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:33.023193  7346 tablet_bootstrap.cc:654] T 7f2519b0c6404799ad84d120b45a7ea4 P a2c90ee440b44b708bccc2e2d5385cc0: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:33.025158  7346 tablet_bootstrap.cc:492] T 7f2519b0c6404799ad84d120b45a7ea4 P a2c90ee440b44b708bccc2e2d5385cc0: No bootstrap required, opened a new log
I20240502 01:55:33.025326  7346 ts_tablet_manager.cc:1367] T 7f2519b0c6404799ad84d120b45a7ea4 P a2c90ee440b44b708bccc2e2d5385cc0: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240502 01:55:33.025758  6316 tablet_service.cc:1467] Processing CreateTablet for tablet 866f87ea9f7f48edb667b1b2eb3fa239 (DEFAULT_TABLE table=rebalance_test_table_3 [id=63a163be603345428cc84c86b96467f0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:33.026096  7346 raft_consensus.cc:348] T 7f2519b0c6404799ad84d120b45a7ea4 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:33.026263  6316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 866f87ea9f7f48edb667b1b2eb3fa239. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:33.026273  7346 raft_consensus.cc:374] T 7f2519b0c6404799ad84d120b45a7ea4 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:33.026355  7346 raft_consensus.cc:729] T 7f2519b0c6404799ad84d120b45a7ea4 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c90ee440b44b708bccc2e2d5385cc0, State: Initialized, Role: FOLLOWER
I20240502 01:55:33.026614  7346 consensus_queue.cc:260] T 7f2519b0c6404799ad84d120b45a7ea4 P a2c90ee440b44b708bccc2e2d5385cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:33.027070  7346 ts_tablet_manager.cc:1398] T 7f2519b0c6404799ad84d120b45a7ea4 P a2c90ee440b44b708bccc2e2d5385cc0: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:55:33.027374  7346 tablet_bootstrap.cc:492] T 866f87ea9f7f48edb667b1b2eb3fa239 P a2c90ee440b44b708bccc2e2d5385cc0: Bootstrap starting.
I20240502 01:55:33.028854  7349 tablet_bootstrap.cc:492] T 866f87ea9f7f48edb667b1b2eb3fa239 P 07cdb213c77c4ffda1f999b66b725399: Bootstrap starting.
I20240502 01:55:33.029081  7346 tablet_bootstrap.cc:654] T 866f87ea9f7f48edb667b1b2eb3fa239 P a2c90ee440b44b708bccc2e2d5385cc0: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:33.030797  7349 tablet_bootstrap.cc:654] T 866f87ea9f7f48edb667b1b2eb3fa239 P 07cdb213c77c4ffda1f999b66b725399: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:33.035207  7345 tablet_bootstrap.cc:492] T 7f2519b0c6404799ad84d120b45a7ea4 P 218bc8c1f9ce46059898fc738179adf9: No bootstrap required, opened a new log
I20240502 01:55:33.035455  7345 ts_tablet_manager.cc:1367] T 7f2519b0c6404799ad84d120b45a7ea4 P 218bc8c1f9ce46059898fc738179adf9: Time spent bootstrapping tablet: real 0.019s	user 0.004s	sys 0.000s
I20240502 01:55:33.036767  7345 raft_consensus.cc:348] T 7f2519b0c6404799ad84d120b45a7ea4 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:33.037045  7345 raft_consensus.cc:374] T 7f2519b0c6404799ad84d120b45a7ea4 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:33.037775  7345 raft_consensus.cc:729] T 7f2519b0c6404799ad84d120b45a7ea4 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218bc8c1f9ce46059898fc738179adf9, State: Initialized, Role: FOLLOWER
I20240502 01:55:33.037840  7346 tablet_bootstrap.cc:492] T 866f87ea9f7f48edb667b1b2eb3fa239 P a2c90ee440b44b708bccc2e2d5385cc0: No bootstrap required, opened a new log
I20240502 01:55:33.038111  7346 ts_tablet_manager.cc:1367] T 866f87ea9f7f48edb667b1b2eb3fa239 P a2c90ee440b44b708bccc2e2d5385cc0: Time spent bootstrapping tablet: real 0.011s	user 0.003s	sys 0.007s
I20240502 01:55:33.038156  7345 consensus_queue.cc:260] T 7f2519b0c6404799ad84d120b45a7ea4 P 218bc8c1f9ce46059898fc738179adf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:33.038937  7345 ts_tablet_manager.cc:1398] T 7f2519b0c6404799ad84d120b45a7ea4 P 218bc8c1f9ce46059898fc738179adf9: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:55:33.039719  7346 raft_consensus.cc:348] T 866f87ea9f7f48edb667b1b2eb3fa239 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:33.040071  7346 raft_consensus.cc:374] T 866f87ea9f7f48edb667b1b2eb3fa239 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:33.040187  7346 raft_consensus.cc:729] T 866f87ea9f7f48edb667b1b2eb3fa239 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c90ee440b44b708bccc2e2d5385cc0, State: Initialized, Role: FOLLOWER
I20240502 01:55:33.040539  7346 consensus_queue.cc:260] T 866f87ea9f7f48edb667b1b2eb3fa239 P a2c90ee440b44b708bccc2e2d5385cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:33.041018  7345 tablet_bootstrap.cc:492] T 6984d638991442e3adaed1b24bee95e9 P 218bc8c1f9ce46059898fc738179adf9: Bootstrap starting.
I20240502 01:55:33.041927  7346 ts_tablet_manager.cc:1398] T 866f87ea9f7f48edb667b1b2eb3fa239 P a2c90ee440b44b708bccc2e2d5385cc0: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 01:55:33.045513  7345 tablet_bootstrap.cc:654] T 6984d638991442e3adaed1b24bee95e9 P 218bc8c1f9ce46059898fc738179adf9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:33.050745  7345 tablet_bootstrap.cc:492] T 6984d638991442e3adaed1b24bee95e9 P 218bc8c1f9ce46059898fc738179adf9: No bootstrap required, opened a new log
I20240502 01:55:33.051095  7345 ts_tablet_manager.cc:1367] T 6984d638991442e3adaed1b24bee95e9 P 218bc8c1f9ce46059898fc738179adf9: Time spent bootstrapping tablet: real 0.012s	user 0.002s	sys 0.005s
I20240502 01:55:33.052194  7345 raft_consensus.cc:348] T 6984d638991442e3adaed1b24bee95e9 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:33.052567  7345 raft_consensus.cc:374] T 6984d638991442e3adaed1b24bee95e9 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:33.052706  7345 raft_consensus.cc:729] T 6984d638991442e3adaed1b24bee95e9 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218bc8c1f9ce46059898fc738179adf9, State: Initialized, Role: FOLLOWER
I20240502 01:55:33.053092  7345 consensus_queue.cc:260] T 6984d638991442e3adaed1b24bee95e9 P 218bc8c1f9ce46059898fc738179adf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:33.053802  7345 ts_tablet_manager.cc:1398] T 6984d638991442e3adaed1b24bee95e9 P 218bc8c1f9ce46059898fc738179adf9: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:55:33.062271  7352 tablet_bootstrap.cc:492] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3: Bootstrap starting.
I20240502 01:55:33.063831  7352 tablet_bootstrap.cc:654] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:33.066437  7352 tablet_bootstrap.cc:492] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3: No bootstrap required, opened a new log
I20240502 01:55:33.066490  7349 tablet_bootstrap.cc:492] T 866f87ea9f7f48edb667b1b2eb3fa239 P 07cdb213c77c4ffda1f999b66b725399: No bootstrap required, opened a new log
I20240502 01:55:33.066619  7352 ts_tablet_manager.cc:1367] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.003s
I20240502 01:55:33.066653  7349 ts_tablet_manager.cc:1367] T 866f87ea9f7f48edb667b1b2eb3fa239 P 07cdb213c77c4ffda1f999b66b725399: Time spent bootstrapping tablet: real 0.038s	user 0.022s	sys 0.011s
I20240502 01:55:33.067373  7349 raft_consensus.cc:348] T 866f87ea9f7f48edb667b1b2eb3fa239 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:33.067431  7352 raft_consensus.cc:348] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:33.067672  7352 raft_consensus.cc:374] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:33.067780  7352 raft_consensus.cc:729] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94ffd7aaf85b4e17881241b8396fa2f3, State: Initialized, Role: FOLLOWER
I20240502 01:55:33.067556  7349 raft_consensus.cc:374] T 866f87ea9f7f48edb667b1b2eb3fa239 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:33.068063  7352 consensus_queue.cc:260] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:33.068775  7352 ts_tablet_manager.cc:1398] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 01:55:33.069002  7349 raft_consensus.cc:729] T 866f87ea9f7f48edb667b1b2eb3fa239 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07cdb213c77c4ffda1f999b66b725399, State: Initialized, Role: FOLLOWER
I20240502 01:55:33.069098  7352 tablet_bootstrap.cc:492] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3: Bootstrap starting.
I20240502 01:55:33.069454  7349 consensus_queue.cc:260] T 866f87ea9f7f48edb667b1b2eb3fa239 P 07cdb213c77c4ffda1f999b66b725399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:33.070138  7349 ts_tablet_manager.cc:1398] T 866f87ea9f7f48edb667b1b2eb3fa239 P 07cdb213c77c4ffda1f999b66b725399: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.001s
I20240502 01:55:33.070474  7349 tablet_bootstrap.cc:492] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399: Bootstrap starting.
I20240502 01:55:33.070987  7352 tablet_bootstrap.cc:654] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:33.076373  7349 tablet_bootstrap.cc:654] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:33.076754  7352 tablet_bootstrap.cc:492] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3: No bootstrap required, opened a new log
I20240502 01:55:33.076930  7352 ts_tablet_manager.cc:1367] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3: Time spent bootstrapping tablet: real 0.008s	user 0.005s	sys 0.001s
I20240502 01:55:33.077664  7352 raft_consensus.cc:348] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:33.077999  7352 raft_consensus.cc:374] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:33.078137  7352 raft_consensus.cc:729] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94ffd7aaf85b4e17881241b8396fa2f3, State: Initialized, Role: FOLLOWER
I20240502 01:55:33.078495  7352 consensus_queue.cc:260] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:33.079150  7352 ts_tablet_manager.cc:1398] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:55:33.079535  7352 tablet_bootstrap.cc:492] T 6984d638991442e3adaed1b24bee95e9 P 94ffd7aaf85b4e17881241b8396fa2f3: Bootstrap starting.
I20240502 01:55:33.081243  7352 tablet_bootstrap.cc:654] T 6984d638991442e3adaed1b24bee95e9 P 94ffd7aaf85b4e17881241b8396fa2f3: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:33.086452  7349 tablet_bootstrap.cc:492] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399: No bootstrap required, opened a new log
I20240502 01:55:33.086679  7349 ts_tablet_manager.cc:1367] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399: Time spent bootstrapping tablet: real 0.016s	user 0.006s	sys 0.003s
I20240502 01:55:33.087529  7349 raft_consensus.cc:348] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:33.087988  7349 raft_consensus.cc:374] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:33.088129  7349 raft_consensus.cc:729] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07cdb213c77c4ffda1f999b66b725399, State: Initialized, Role: FOLLOWER
I20240502 01:55:33.088490  7349 consensus_queue.cc:260] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:33.089216  7349 ts_tablet_manager.cc:1398] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
I20240502 01:55:33.089763  7352 tablet_bootstrap.cc:492] T 6984d638991442e3adaed1b24bee95e9 P 94ffd7aaf85b4e17881241b8396fa2f3: No bootstrap required, opened a new log
I20240502 01:55:33.089924  7352 ts_tablet_manager.cc:1367] T 6984d638991442e3adaed1b24bee95e9 P 94ffd7aaf85b4e17881241b8396fa2f3: Time spent bootstrapping tablet: real 0.010s	user 0.009s	sys 0.000s
I20240502 01:55:33.090615  7352 raft_consensus.cc:348] T 6984d638991442e3adaed1b24bee95e9 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:33.090796  7352 raft_consensus.cc:374] T 6984d638991442e3adaed1b24bee95e9 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:33.090888  7352 raft_consensus.cc:729] T 6984d638991442e3adaed1b24bee95e9 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94ffd7aaf85b4e17881241b8396fa2f3, State: Initialized, Role: FOLLOWER
I20240502 01:55:33.091120  7352 consensus_queue.cc:260] T 6984d638991442e3adaed1b24bee95e9 P 94ffd7aaf85b4e17881241b8396fa2f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:33.091730  7352 ts_tablet_manager.cc:1398] T 6984d638991442e3adaed1b24bee95e9 P 94ffd7aaf85b4e17881241b8396fa2f3: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:55:33.096735  7282 raft_consensus.cc:482] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:33.096899  7282 raft_consensus.cc:504] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:33.097546  7282 leader_election.cc:288] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867), a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197)
I20240502 01:55:33.098094  6469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "866f87ea9f7f48edb667b1b2eb3fa239" candidate_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07cdb213c77c4ffda1f999b66b725399" is_pre_election: true
I20240502 01:55:33.098127  6203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "866f87ea9f7f48edb667b1b2eb3fa239" candidate_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" is_pre_election: true
I20240502 01:55:33.098352  6469 raft_consensus.cc:2454] T 866f87ea9f7f48edb667b1b2eb3fa239 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94ffd7aaf85b4e17881241b8396fa2f3 in term 0.
I20240502 01:55:33.098409  6203 raft_consensus.cc:2454] T 866f87ea9f7f48edb667b1b2eb3fa239 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94ffd7aaf85b4e17881241b8396fa2f3 in term 0.
I20240502 01:55:33.098870  6268 leader_election.cc:302] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 07cdb213c77c4ffda1f999b66b725399, 94ffd7aaf85b4e17881241b8396fa2f3, a2c90ee440b44b708bccc2e2d5385cc0; no voters: 
I20240502 01:55:33.099185  7282 raft_consensus.cc:2781] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:33.099314  7282 raft_consensus.cc:482] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:33.099406  7282 raft_consensus.cc:3037] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:33.101572  7282 raft_consensus.cc:504] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:33.102257  7282 leader_election.cc:288] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 1 election: Requested vote from peers 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867), a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197)
I20240502 01:55:33.102820  6469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "866f87ea9f7f48edb667b1b2eb3fa239" candidate_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07cdb213c77c4ffda1f999b66b725399"
I20240502 01:55:33.102819  6203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "866f87ea9f7f48edb667b1b2eb3fa239" candidate_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c90ee440b44b708bccc2e2d5385cc0"
I20240502 01:55:33.103057  6469 raft_consensus.cc:3037] T 866f87ea9f7f48edb667b1b2eb3fa239 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:33.103080  6203 raft_consensus.cc:3037] T 866f87ea9f7f48edb667b1b2eb3fa239 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:33.105993  6203 raft_consensus.cc:2454] T 866f87ea9f7f48edb667b1b2eb3fa239 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ffd7aaf85b4e17881241b8396fa2f3 in term 1.
I20240502 01:55:33.106005  6469 raft_consensus.cc:2454] T 866f87ea9f7f48edb667b1b2eb3fa239 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ffd7aaf85b4e17881241b8396fa2f3 in term 1.
I20240502 01:55:33.106508  6271 leader_election.cc:302] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94ffd7aaf85b4e17881241b8396fa2f3, a2c90ee440b44b708bccc2e2d5385cc0; no voters: 
I20240502 01:55:33.106911  7282 raft_consensus.cc:2781] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:33.107089  7282 raft_consensus.cc:686] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Becoming Leader. State: Replica: 94ffd7aaf85b4e17881241b8396fa2f3, State: Running, Role: LEADER
I20240502 01:55:33.107398  7282 consensus_queue.cc:237] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:33.110538  5913 catalog_manager.cc:5484] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131). New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } }
I20240502 01:55:33.140394  7282 raft_consensus.cc:482] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:33.140621  7282 raft_consensus.cc:504] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:33.141284  7282 leader_election.cc:288] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227), a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197)
I20240502 01:55:33.141824  6068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f2519b0c6404799ad84d120b45a7ea4" candidate_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218bc8c1f9ce46059898fc738179adf9" is_pre_election: true
I20240502 01:55:33.141868  6203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f2519b0c6404799ad84d120b45a7ea4" candidate_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" is_pre_election: true
I20240502 01:55:33.142068  6068 raft_consensus.cc:2454] T 7f2519b0c6404799ad84d120b45a7ea4 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94ffd7aaf85b4e17881241b8396fa2f3 in term 0.
I20240502 01:55:33.142091  6203 raft_consensus.cc:2454] T 7f2519b0c6404799ad84d120b45a7ea4 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94ffd7aaf85b4e17881241b8396fa2f3 in term 0.
I20240502 01:55:33.142493  6270 leader_election.cc:302] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9, 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 
I20240502 01:55:33.142757  7282 raft_consensus.cc:2781] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:33.142846  7282 raft_consensus.cc:482] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:33.142944  7282 raft_consensus.cc:3037] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:33.143468  7254 raft_consensus.cc:482] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:33.143635  7254 raft_consensus.cc:504] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:33.144395  7254 leader_election.cc:288] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245), 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227)
I20240502 01:55:33.144883  6335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6984d638991442e3adaed1b24bee95e9" candidate_uuid: "07cdb213c77c4ffda1f999b66b725399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" is_pre_election: true
I20240502 01:55:33.145046  6068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6984d638991442e3adaed1b24bee95e9" candidate_uuid: "07cdb213c77c4ffda1f999b66b725399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218bc8c1f9ce46059898fc738179adf9" is_pre_election: true
I20240502 01:55:33.145148  6335 raft_consensus.cc:2454] T 6984d638991442e3adaed1b24bee95e9 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07cdb213c77c4ffda1f999b66b725399 in term 0.
I20240502 01:55:33.145268  6068 raft_consensus.cc:2454] T 6984d638991442e3adaed1b24bee95e9 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07cdb213c77c4ffda1f999b66b725399 in term 0.
I20240502 01:55:33.145624  6402 leader_election.cc:302] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07cdb213c77c4ffda1f999b66b725399, 218bc8c1f9ce46059898fc738179adf9; no voters: 
I20240502 01:55:33.145567  7282 raft_consensus.cc:504] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:33.145895  7254 raft_consensus.cc:2781] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:33.146026  7254 raft_consensus.cc:482] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:33.146147  7254 raft_consensus.cc:3037] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:33.146171  7282 leader_election.cc:288] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 1 election: Requested vote from peers 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227), a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197)
I20240502 01:55:33.146569  6068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f2519b0c6404799ad84d120b45a7ea4" candidate_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218bc8c1f9ce46059898fc738179adf9"
I20240502 01:55:33.146771  6068 raft_consensus.cc:3037] T 7f2519b0c6404799ad84d120b45a7ea4 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:33.146843  6203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f2519b0c6404799ad84d120b45a7ea4" candidate_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c90ee440b44b708bccc2e2d5385cc0"
I20240502 01:55:33.147009  6203 raft_consensus.cc:3037] T 7f2519b0c6404799ad84d120b45a7ea4 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:33.148902  6203 raft_consensus.cc:2454] T 7f2519b0c6404799ad84d120b45a7ea4 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ffd7aaf85b4e17881241b8396fa2f3 in term 1.
I20240502 01:55:33.148916  7254 raft_consensus.cc:504] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:33.149360  6271 leader_election.cc:302] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94ffd7aaf85b4e17881241b8396fa2f3, a2c90ee440b44b708bccc2e2d5385cc0; no voters: 
I20240502 01:55:33.149477  6068 raft_consensus.cc:2454] T 7f2519b0c6404799ad84d120b45a7ea4 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ffd7aaf85b4e17881241b8396fa2f3 in term 1.
I20240502 01:55:33.149646  7282 raft_consensus.cc:2781] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:33.149915  7282 raft_consensus.cc:686] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Becoming Leader. State: Replica: 94ffd7aaf85b4e17881241b8396fa2f3, State: Running, Role: LEADER
I20240502 01:55:33.150230  7254 leader_election.cc:288] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 [CANDIDATE]: Term 1 election: Requested vote from peers 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245), 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227)
I20240502 01:55:33.150238  6335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6984d638991442e3adaed1b24bee95e9" candidate_uuid: "07cdb213c77c4ffda1f999b66b725399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ffd7aaf85b4e17881241b8396fa2f3"
I20240502 01:55:33.150210  7282 consensus_queue.cc:237] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:33.150445  6335 raft_consensus.cc:3037] T 6984d638991442e3adaed1b24bee95e9 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:33.150384  6068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6984d638991442e3adaed1b24bee95e9" candidate_uuid: "07cdb213c77c4ffda1f999b66b725399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218bc8c1f9ce46059898fc738179adf9"
I20240502 01:55:33.150568  6068 raft_consensus.cc:3037] T 6984d638991442e3adaed1b24bee95e9 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:33.152904  6068 raft_consensus.cc:2454] T 6984d638991442e3adaed1b24bee95e9 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07cdb213c77c4ffda1f999b66b725399 in term 1.
I20240502 01:55:33.153194  6335 raft_consensus.cc:2454] T 6984d638991442e3adaed1b24bee95e9 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07cdb213c77c4ffda1f999b66b725399 in term 1.
I20240502 01:55:33.153439  6402 leader_election.cc:302] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07cdb213c77c4ffda1f999b66b725399, 218bc8c1f9ce46059898fc738179adf9; no voters: 
I20240502 01:55:33.153779  7254 raft_consensus.cc:2781] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:33.154062  7254 raft_consensus.cc:686] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Becoming Leader. State: Replica: 07cdb213c77c4ffda1f999b66b725399, State: Running, Role: LEADER
I20240502 01:55:33.154387  7254 consensus_queue.cc:237] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:33.155593  5913 catalog_manager.cc:5484] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131). New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } }
I20240502 01:55:33.157955  5916 catalog_manager.cc:5484] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132). New cstate: current_term: 1 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } }
W20240502 01:55:33.535290  7361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:33.535655  7361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:33.543998  7361 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:55:33.548918  7086 consensus_queue.cc:1035] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:33.560612  7086 consensus_queue.cc:1035] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
I20240502 01:55:33.676543  7254 consensus_queue.cc:1035] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:33.694262  7393 consensus_queue.cc:1035] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:33.703555  7282 consensus_queue.cc:1035] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:33.731767  7086 consensus_queue.cc:1035] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:55:33.816730  6270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.129:36227, user_credentials={real_user=slave}} blocked reactor thread for 45331.2us

INSERT report
    rows total: 2000
    time total: 297.557 ms
  time per row: 0.148779 ms
I20240502 01:55:34.009025  5917 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58156:
name: "rebalance_test_table_4"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:55:34.020337  6046 tablet_service.cc:1467] Processing CreateTablet for tablet 74eadf4c939c4c6383732cd74c5ec9d6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1ed4a88787d6401481245c552170a456]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:34.020342  6181 tablet_service.cc:1467] Processing CreateTablet for tablet 74eadf4c939c4c6383732cd74c5ec9d6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1ed4a88787d6401481245c552170a456]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:34.020855  6046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74eadf4c939c4c6383732cd74c5ec9d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:34.020855  6181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74eadf4c939c4c6383732cd74c5ec9d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:34.023619  6315 tablet_service.cc:1467] Processing CreateTablet for tablet 74eadf4c939c4c6383732cd74c5ec9d6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1ed4a88787d6401481245c552170a456]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:34.024128  6315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74eadf4c939c4c6383732cd74c5ec9d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:34.026175  6047 tablet_service.cc:1467] Processing CreateTablet for tablet c58d6b2401c047cd9af56a66c2e00105 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1ed4a88787d6401481245c552170a456]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:34.026574  6446 tablet_service.cc:1467] Processing CreateTablet for tablet c58d6b2401c047cd9af56a66c2e00105 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1ed4a88787d6401481245c552170a456]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:34.026706  6048 tablet_service.cc:1467] Processing CreateTablet for tablet 55de74ae85e541a19008fd2fc2da3ffd (DEFAULT_TABLE table=rebalance_test_table_4 [id=1ed4a88787d6401481245c552170a456]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:34.027002  6446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c58d6b2401c047cd9af56a66c2e00105. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:34.027498  6180 tablet_service.cc:1467] Processing CreateTablet for tablet c58d6b2401c047cd9af56a66c2e00105 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1ed4a88787d6401481245c552170a456]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:34.027966  6180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c58d6b2401c047cd9af56a66c2e00105. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:34.032277  7427 tablet_bootstrap.cc:492] T 74eadf4c939c4c6383732cd74c5ec9d6 P 218bc8c1f9ce46059898fc738179adf9: Bootstrap starting.
I20240502 01:55:34.032354  6447 tablet_service.cc:1467] Processing CreateTablet for tablet 55de74ae85e541a19008fd2fc2da3ffd (DEFAULT_TABLE table=rebalance_test_table_4 [id=1ed4a88787d6401481245c552170a456]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:34.032775  6447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55de74ae85e541a19008fd2fc2da3ffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:34.033071  6314 tablet_service.cc:1467] Processing CreateTablet for tablet 55de74ae85e541a19008fd2fc2da3ffd (DEFAULT_TABLE table=rebalance_test_table_4 [id=1ed4a88787d6401481245c552170a456]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:34.026706  6047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c58d6b2401c047cd9af56a66c2e00105. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:34.033522  6314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55de74ae85e541a19008fd2fc2da3ffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:34.033748  6048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55de74ae85e541a19008fd2fc2da3ffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:34.043119  7429 tablet_bootstrap.cc:492] T 55de74ae85e541a19008fd2fc2da3ffd P 94ffd7aaf85b4e17881241b8396fa2f3: Bootstrap starting.
I20240502 01:55:34.047217  7429 tablet_bootstrap.cc:654] T 55de74ae85e541a19008fd2fc2da3ffd P 94ffd7aaf85b4e17881241b8396fa2f3: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:34.054487  7428 tablet_bootstrap.cc:492] T 55de74ae85e541a19008fd2fc2da3ffd P 07cdb213c77c4ffda1f999b66b725399: Bootstrap starting.
I20240502 01:55:34.056434  7427 tablet_bootstrap.cc:654] T 74eadf4c939c4c6383732cd74c5ec9d6 P 218bc8c1f9ce46059898fc738179adf9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:34.058401  7430 tablet_bootstrap.cc:492] T 74eadf4c939c4c6383732cd74c5ec9d6 P a2c90ee440b44b708bccc2e2d5385cc0: Bootstrap starting.
I20240502 01:55:34.059625  7428 tablet_bootstrap.cc:654] T 55de74ae85e541a19008fd2fc2da3ffd P 07cdb213c77c4ffda1f999b66b725399: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:34.060608  7430 tablet_bootstrap.cc:654] T 74eadf4c939c4c6383732cd74c5ec9d6 P a2c90ee440b44b708bccc2e2d5385cc0: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:34.064635  7430 tablet_bootstrap.cc:492] T 74eadf4c939c4c6383732cd74c5ec9d6 P a2c90ee440b44b708bccc2e2d5385cc0: No bootstrap required, opened a new log
I20240502 01:55:34.065569  7430 ts_tablet_manager.cc:1367] T 74eadf4c939c4c6383732cd74c5ec9d6 P a2c90ee440b44b708bccc2e2d5385cc0: Time spent bootstrapping tablet: real 0.007s	user 0.005s	sys 0.000s
I20240502 01:55:34.066949  7429 tablet_bootstrap.cc:492] T 55de74ae85e541a19008fd2fc2da3ffd P 94ffd7aaf85b4e17881241b8396fa2f3: No bootstrap required, opened a new log
I20240502 01:55:34.067296  7429 ts_tablet_manager.cc:1367] T 55de74ae85e541a19008fd2fc2da3ffd P 94ffd7aaf85b4e17881241b8396fa2f3: Time spent bootstrapping tablet: real 0.024s	user 0.018s	sys 0.000s
I20240502 01:55:34.068002  7428 tablet_bootstrap.cc:492] T 55de74ae85e541a19008fd2fc2da3ffd P 07cdb213c77c4ffda1f999b66b725399: No bootstrap required, opened a new log
I20240502 01:55:34.068224  7428 ts_tablet_manager.cc:1367] T 55de74ae85e541a19008fd2fc2da3ffd P 07cdb213c77c4ffda1f999b66b725399: Time spent bootstrapping tablet: real 0.014s	user 0.007s	sys 0.003s
I20240502 01:55:34.068434  7429 raft_consensus.cc:348] T 55de74ae85e541a19008fd2fc2da3ffd P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 01:55:34.068830  7429 raft_consensus.cc:374] T 55de74ae85e541a19008fd2fc2da3ffd P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:34.068979  7429 raft_consensus.cc:729] T 55de74ae85e541a19008fd2fc2da3ffd P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94ffd7aaf85b4e17881241b8396fa2f3, State: Initialized, Role: FOLLOWER
I20240502 01:55:34.069041  7428 raft_consensus.cc:348] T 55de74ae85e541a19008fd2fc2da3ffd P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 01:55:34.069334  7428 raft_consensus.cc:374] T 55de74ae85e541a19008fd2fc2da3ffd P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:34.069453  7428 raft_consensus.cc:729] T 55de74ae85e541a19008fd2fc2da3ffd P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07cdb213c77c4ffda1f999b66b725399, State: Initialized, Role: FOLLOWER
I20240502 01:55:34.069382  7429 consensus_queue.cc:260] T 55de74ae85e541a19008fd2fc2da3ffd P 94ffd7aaf85b4e17881241b8396fa2f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 01:55:34.069751  7428 consensus_queue.cc:260] T 55de74ae85e541a19008fd2fc2da3ffd P 07cdb213c77c4ffda1f999b66b725399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 01:55:34.070441  7429 ts_tablet_manager.cc:1398] T 55de74ae85e541a19008fd2fc2da3ffd P 94ffd7aaf85b4e17881241b8396fa2f3: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:55:34.071453  7428 ts_tablet_manager.cc:1398] T 55de74ae85e541a19008fd2fc2da3ffd P 07cdb213c77c4ffda1f999b66b725399: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:55:34.071810  7428 tablet_bootstrap.cc:492] T c58d6b2401c047cd9af56a66c2e00105 P 07cdb213c77c4ffda1f999b66b725399: Bootstrap starting.
I20240502 01:55:34.072161  7430 raft_consensus.cc:348] T 74eadf4c939c4c6383732cd74c5ec9d6 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:34.070825  7429 tablet_bootstrap.cc:492] T 74eadf4c939c4c6383732cd74c5ec9d6 P 94ffd7aaf85b4e17881241b8396fa2f3: Bootstrap starting.
I20240502 01:55:34.073652  7428 tablet_bootstrap.cc:654] T c58d6b2401c047cd9af56a66c2e00105 P 07cdb213c77c4ffda1f999b66b725399: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:34.074348  7429 tablet_bootstrap.cc:654] T 74eadf4c939c4c6383732cd74c5ec9d6 P 94ffd7aaf85b4e17881241b8396fa2f3: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:34.074534  7430 raft_consensus.cc:374] T 74eadf4c939c4c6383732cd74c5ec9d6 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:34.074712  7430 raft_consensus.cc:729] T 74eadf4c939c4c6383732cd74c5ec9d6 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c90ee440b44b708bccc2e2d5385cc0, State: Initialized, Role: FOLLOWER
I20240502 01:55:34.075130  7430 consensus_queue.cc:260] T 74eadf4c939c4c6383732cd74c5ec9d6 P a2c90ee440b44b708bccc2e2d5385cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:34.076102  7430 ts_tablet_manager.cc:1398] T 74eadf4c939c4c6383732cd74c5ec9d6 P a2c90ee440b44b708bccc2e2d5385cc0: Time spent starting tablet: real 0.010s	user 0.002s	sys 0.000s
I20240502 01:55:34.076570  7430 tablet_bootstrap.cc:492] T c58d6b2401c047cd9af56a66c2e00105 P a2c90ee440b44b708bccc2e2d5385cc0: Bootstrap starting.
I20240502 01:55:34.079758  7430 tablet_bootstrap.cc:654] T c58d6b2401c047cd9af56a66c2e00105 P a2c90ee440b44b708bccc2e2d5385cc0: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:34.085855  7428 tablet_bootstrap.cc:492] T c58d6b2401c047cd9af56a66c2e00105 P 07cdb213c77c4ffda1f999b66b725399: No bootstrap required, opened a new log
I20240502 01:55:34.086186  7428 ts_tablet_manager.cc:1367] T c58d6b2401c047cd9af56a66c2e00105 P 07cdb213c77c4ffda1f999b66b725399: Time spent bootstrapping tablet: real 0.014s	user 0.002s	sys 0.008s
I20240502 01:55:34.087143  7428 raft_consensus.cc:348] T c58d6b2401c047cd9af56a66c2e00105 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:34.087491  7428 raft_consensus.cc:374] T c58d6b2401c047cd9af56a66c2e00105 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:34.087628  7428 raft_consensus.cc:729] T c58d6b2401c047cd9af56a66c2e00105 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07cdb213c77c4ffda1f999b66b725399, State: Initialized, Role: FOLLOWER
I20240502 01:55:34.087888  7427 tablet_bootstrap.cc:492] T 74eadf4c939c4c6383732cd74c5ec9d6 P 218bc8c1f9ce46059898fc738179adf9: No bootstrap required, opened a new log
I20240502 01:55:34.088114  7427 ts_tablet_manager.cc:1367] T 74eadf4c939c4c6383732cd74c5ec9d6 P 218bc8c1f9ce46059898fc738179adf9: Time spent bootstrapping tablet: real 0.056s	user 0.012s	sys 0.014s
I20240502 01:55:34.088078  7428 consensus_queue.cc:260] T c58d6b2401c047cd9af56a66c2e00105 P 07cdb213c77c4ffda1f999b66b725399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:34.088917  7428 ts_tablet_manager.cc:1398] T c58d6b2401c047cd9af56a66c2e00105 P 07cdb213c77c4ffda1f999b66b725399: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 01:55:34.090401  7430 tablet_bootstrap.cc:492] T c58d6b2401c047cd9af56a66c2e00105 P a2c90ee440b44b708bccc2e2d5385cc0: No bootstrap required, opened a new log
I20240502 01:55:34.090665  7430 ts_tablet_manager.cc:1367] T c58d6b2401c047cd9af56a66c2e00105 P a2c90ee440b44b708bccc2e2d5385cc0: Time spent bootstrapping tablet: real 0.014s	user 0.008s	sys 0.000s
I20240502 01:55:34.090421  7427 raft_consensus.cc:348] T 74eadf4c939c4c6383732cd74c5ec9d6 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:34.090878  7427 raft_consensus.cc:374] T 74eadf4c939c4c6383732cd74c5ec9d6 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:34.091243  7427 raft_consensus.cc:729] T 74eadf4c939c4c6383732cd74c5ec9d6 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218bc8c1f9ce46059898fc738179adf9, State: Initialized, Role: FOLLOWER
I20240502 01:55:34.091562  7427 consensus_queue.cc:260] T 74eadf4c939c4c6383732cd74c5ec9d6 P 218bc8c1f9ce46059898fc738179adf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:34.092290  7429 tablet_bootstrap.cc:492] T 74eadf4c939c4c6383732cd74c5ec9d6 P 94ffd7aaf85b4e17881241b8396fa2f3: No bootstrap required, opened a new log
I20240502 01:55:34.092430  7427 ts_tablet_manager.cc:1398] T 74eadf4c939c4c6383732cd74c5ec9d6 P 218bc8c1f9ce46059898fc738179adf9: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 01:55:34.092540  7429 ts_tablet_manager.cc:1367] T 74eadf4c939c4c6383732cd74c5ec9d6 P 94ffd7aaf85b4e17881241b8396fa2f3: Time spent bootstrapping tablet: real 0.022s	user 0.004s	sys 0.005s
I20240502 01:55:34.092783  7427 tablet_bootstrap.cc:492] T 55de74ae85e541a19008fd2fc2da3ffd P 218bc8c1f9ce46059898fc738179adf9: Bootstrap starting.
I20240502 01:55:34.093639  7429 raft_consensus.cc:348] T 74eadf4c939c4c6383732cd74c5ec9d6 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:34.093947  7429 raft_consensus.cc:374] T 74eadf4c939c4c6383732cd74c5ec9d6 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:34.094050  7429 raft_consensus.cc:729] T 74eadf4c939c4c6383732cd74c5ec9d6 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94ffd7aaf85b4e17881241b8396fa2f3, State: Initialized, Role: FOLLOWER
I20240502 01:55:34.094388  7429 consensus_queue.cc:260] T 74eadf4c939c4c6383732cd74c5ec9d6 P 94ffd7aaf85b4e17881241b8396fa2f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:34.094667  7427 tablet_bootstrap.cc:654] T 55de74ae85e541a19008fd2fc2da3ffd P 218bc8c1f9ce46059898fc738179adf9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:34.095497  7429 ts_tablet_manager.cc:1398] T 74eadf4c939c4c6383732cd74c5ec9d6 P 94ffd7aaf85b4e17881241b8396fa2f3: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:55:34.096333  7430 raft_consensus.cc:348] T c58d6b2401c047cd9af56a66c2e00105 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:34.096658  7430 raft_consensus.cc:374] T c58d6b2401c047cd9af56a66c2e00105 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:34.099339  7430 raft_consensus.cc:729] T c58d6b2401c047cd9af56a66c2e00105 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c90ee440b44b708bccc2e2d5385cc0, State: Initialized, Role: FOLLOWER
I20240502 01:55:34.099691  7430 consensus_queue.cc:260] T c58d6b2401c047cd9af56a66c2e00105 P a2c90ee440b44b708bccc2e2d5385cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:34.100478  7430 ts_tablet_manager.cc:1398] T c58d6b2401c047cd9af56a66c2e00105 P a2c90ee440b44b708bccc2e2d5385cc0: Time spent starting tablet: real 0.010s	user 0.002s	sys 0.000s
I20240502 01:55:34.102902  7427 tablet_bootstrap.cc:492] T 55de74ae85e541a19008fd2fc2da3ffd P 218bc8c1f9ce46059898fc738179adf9: No bootstrap required, opened a new log
I20240502 01:55:34.103160  7427 ts_tablet_manager.cc:1367] T 55de74ae85e541a19008fd2fc2da3ffd P 218bc8c1f9ce46059898fc738179adf9: Time spent bootstrapping tablet: real 0.010s	user 0.004s	sys 0.004s
I20240502 01:55:34.104297  7427 raft_consensus.cc:348] T 55de74ae85e541a19008fd2fc2da3ffd P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 01:55:34.104496  7427 raft_consensus.cc:374] T 55de74ae85e541a19008fd2fc2da3ffd P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:34.104574  7427 raft_consensus.cc:729] T 55de74ae85e541a19008fd2fc2da3ffd P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218bc8c1f9ce46059898fc738179adf9, State: Initialized, Role: FOLLOWER
I20240502 01:55:34.104840  7427 consensus_queue.cc:260] T 55de74ae85e541a19008fd2fc2da3ffd P 218bc8c1f9ce46059898fc738179adf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 01:55:34.105505  7427 ts_tablet_manager.cc:1398] T 55de74ae85e541a19008fd2fc2da3ffd P 218bc8c1f9ce46059898fc738179adf9: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 01:55:34.105865  7427 tablet_bootstrap.cc:492] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9: Bootstrap starting.
I20240502 01:55:34.107270  7427 tablet_bootstrap.cc:654] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:34.109299  7427 tablet_bootstrap.cc:492] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9: No bootstrap required, opened a new log
I20240502 01:55:34.109452  7427 ts_tablet_manager.cc:1367] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.003s
I20240502 01:55:34.110119  7427 raft_consensus.cc:348] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:34.110286  7427 raft_consensus.cc:374] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:34.110353  7427 raft_consensus.cc:729] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218bc8c1f9ce46059898fc738179adf9, State: Initialized, Role: FOLLOWER
I20240502 01:55:34.110553  7427 consensus_queue.cc:260] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:34.111027  7427 ts_tablet_manager.cc:1398] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 01:55:34.148700  7409 raft_consensus.cc:482] T 55de74ae85e541a19008fd2fc2da3ffd P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:34.148928  7409 raft_consensus.cc:504] T 55de74ae85e541a19008fd2fc2da3ffd P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 01:55:34.149704  7409 leader_election.cc:288] T 55de74ae85e541a19008fd2fc2da3ffd P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867), 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227)
I20240502 01:55:34.150274  6068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55de74ae85e541a19008fd2fc2da3ffd" candidate_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218bc8c1f9ce46059898fc738179adf9" is_pre_election: true
I20240502 01:55:34.150288  6469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55de74ae85e541a19008fd2fc2da3ffd" candidate_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07cdb213c77c4ffda1f999b66b725399" is_pre_election: true
I20240502 01:55:34.150532  6068 raft_consensus.cc:2454] T 55de74ae85e541a19008fd2fc2da3ffd P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94ffd7aaf85b4e17881241b8396fa2f3 in term 0.
I20240502 01:55:34.150533  6469 raft_consensus.cc:2454] T 55de74ae85e541a19008fd2fc2da3ffd P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94ffd7aaf85b4e17881241b8396fa2f3 in term 0.
I20240502 01:55:34.151053  6268 leader_election.cc:302] T 55de74ae85e541a19008fd2fc2da3ffd P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07cdb213c77c4ffda1f999b66b725399, 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 
I20240502 01:55:34.151374  7409 raft_consensus.cc:2781] T 55de74ae85e541a19008fd2fc2da3ffd P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:34.151494  7409 raft_consensus.cc:482] T 55de74ae85e541a19008fd2fc2da3ffd P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:34.151567  7409 raft_consensus.cc:3037] T 55de74ae85e541a19008fd2fc2da3ffd P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:34.153606  7409 raft_consensus.cc:504] T 55de74ae85e541a19008fd2fc2da3ffd P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 01:55:34.154314  7409 leader_election.cc:288] T 55de74ae85e541a19008fd2fc2da3ffd P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 1 election: Requested vote from peers 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867), 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227)
I20240502 01:55:34.154886  6469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55de74ae85e541a19008fd2fc2da3ffd" candidate_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07cdb213c77c4ffda1f999b66b725399"
I20240502 01:55:34.154920  6068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55de74ae85e541a19008fd2fc2da3ffd" candidate_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218bc8c1f9ce46059898fc738179adf9"
I20240502 01:55:34.155155  6469 raft_consensus.cc:3037] T 55de74ae85e541a19008fd2fc2da3ffd P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:34.155169  6068 raft_consensus.cc:3037] T 55de74ae85e541a19008fd2fc2da3ffd P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:34.158133  6068 raft_consensus.cc:2454] T 55de74ae85e541a19008fd2fc2da3ffd P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ffd7aaf85b4e17881241b8396fa2f3 in term 1.
I20240502 01:55:34.158133  6469 raft_consensus.cc:2454] T 55de74ae85e541a19008fd2fc2da3ffd P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ffd7aaf85b4e17881241b8396fa2f3 in term 1.
I20240502 01:55:34.158731  6268 leader_election.cc:302] T 55de74ae85e541a19008fd2fc2da3ffd P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 07cdb213c77c4ffda1f999b66b725399, 218bc8c1f9ce46059898fc738179adf9, 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 
I20240502 01:55:34.159134  7409 raft_consensus.cc:2781] T 55de74ae85e541a19008fd2fc2da3ffd P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:34.159300  7409 raft_consensus.cc:686] T 55de74ae85e541a19008fd2fc2da3ffd P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Becoming Leader. State: Replica: 94ffd7aaf85b4e17881241b8396fa2f3, State: Running, Role: LEADER
I20240502 01:55:34.159600  7409 consensus_queue.cc:237] T 55de74ae85e541a19008fd2fc2da3ffd P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 01:55:34.162127  7409 raft_consensus.cc:482] T 74eadf4c939c4c6383732cd74c5ec9d6 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:34.162294  7409 raft_consensus.cc:504] T 74eadf4c939c4c6383732cd74c5ec9d6 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:34.162673  5912 catalog_manager.cc:5484] T 55de74ae85e541a19008fd2fc2da3ffd P 94ffd7aaf85b4e17881241b8396fa2f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131). New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } }
I20240502 01:55:34.162945  7409 leader_election.cc:288] T 74eadf4c939c4c6383732cd74c5ec9d6 P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227), a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197)
I20240502 01:55:34.163592  6203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74eadf4c939c4c6383732cd74c5ec9d6" candidate_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" is_pre_election: true
I20240502 01:55:34.163897  6203 raft_consensus.cc:2454] T 74eadf4c939c4c6383732cd74c5ec9d6 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94ffd7aaf85b4e17881241b8396fa2f3 in term 0.
I20240502 01:55:34.164083  6068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74eadf4c939c4c6383732cd74c5ec9d6" candidate_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218bc8c1f9ce46059898fc738179adf9" is_pre_election: true
I20240502 01:55:34.164327  6068 raft_consensus.cc:2454] T 74eadf4c939c4c6383732cd74c5ec9d6 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94ffd7aaf85b4e17881241b8396fa2f3 in term 0.
I20240502 01:55:34.164703  6271 leader_election.cc:302] T 74eadf4c939c4c6383732cd74c5ec9d6 P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94ffd7aaf85b4e17881241b8396fa2f3, a2c90ee440b44b708bccc2e2d5385cc0; no voters: 
I20240502 01:55:34.165234  7409 raft_consensus.cc:2781] T 74eadf4c939c4c6383732cd74c5ec9d6 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:34.165396  7409 raft_consensus.cc:482] T 74eadf4c939c4c6383732cd74c5ec9d6 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:34.165530  7409 raft_consensus.cc:3037] T 74eadf4c939c4c6383732cd74c5ec9d6 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:34.168030  7409 raft_consensus.cc:504] T 74eadf4c939c4c6383732cd74c5ec9d6 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:34.168717  7409 leader_election.cc:288] T 74eadf4c939c4c6383732cd74c5ec9d6 P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 1 election: Requested vote from peers 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227), a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197)
I20240502 01:55:34.169248  6068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74eadf4c939c4c6383732cd74c5ec9d6" candidate_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218bc8c1f9ce46059898fc738179adf9"
I20240502 01:55:34.169301  6203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74eadf4c939c4c6383732cd74c5ec9d6" candidate_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c90ee440b44b708bccc2e2d5385cc0"
I20240502 01:55:34.169512  6068 raft_consensus.cc:3037] T 74eadf4c939c4c6383732cd74c5ec9d6 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:34.169538  6203 raft_consensus.cc:3037] T 74eadf4c939c4c6383732cd74c5ec9d6 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:34.172505  6068 raft_consensus.cc:2454] T 74eadf4c939c4c6383732cd74c5ec9d6 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ffd7aaf85b4e17881241b8396fa2f3 in term 1.
I20240502 01:55:34.172492  6203 raft_consensus.cc:2454] T 74eadf4c939c4c6383732cd74c5ec9d6 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ffd7aaf85b4e17881241b8396fa2f3 in term 1.
I20240502 01:55:34.173097  6270 leader_election.cc:302] T 74eadf4c939c4c6383732cd74c5ec9d6 P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9, 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 
I20240502 01:55:34.173487  7409 raft_consensus.cc:2781] T 74eadf4c939c4c6383732cd74c5ec9d6 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:34.173660  7409 raft_consensus.cc:686] T 74eadf4c939c4c6383732cd74c5ec9d6 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Becoming Leader. State: Replica: 94ffd7aaf85b4e17881241b8396fa2f3, State: Running, Role: LEADER
I20240502 01:55:34.173959  7409 consensus_queue.cc:237] T 74eadf4c939c4c6383732cd74c5ec9d6 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:34.176879  5912 catalog_manager.cc:5484] T 74eadf4c939c4c6383732cd74c5ec9d6 P 94ffd7aaf85b4e17881241b8396fa2f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131). New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } }
I20240502 01:55:34.328343  7371 raft_consensus.cc:482] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:34.328541  7371 raft_consensus.cc:504] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:34.330137  7371 leader_election.cc:288] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867), a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197)
I20240502 01:55:34.338829  6469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c58d6b2401c047cd9af56a66c2e00105" candidate_uuid: "218bc8c1f9ce46059898fc738179adf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07cdb213c77c4ffda1f999b66b725399" is_pre_election: true
I20240502 01:55:34.338932  6203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c58d6b2401c047cd9af56a66c2e00105" candidate_uuid: "218bc8c1f9ce46059898fc738179adf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" is_pre_election: true
I20240502 01:55:34.339075  6469 raft_consensus.cc:2454] T c58d6b2401c047cd9af56a66c2e00105 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 218bc8c1f9ce46059898fc738179adf9 in term 0.
I20240502 01:55:34.339183  6203 raft_consensus.cc:2454] T c58d6b2401c047cd9af56a66c2e00105 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 218bc8c1f9ce46059898fc738179adf9 in term 0.
I20240502 01:55:34.339629  5998 leader_election.cc:302] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 07cdb213c77c4ffda1f999b66b725399, 218bc8c1f9ce46059898fc738179adf9, a2c90ee440b44b708bccc2e2d5385cc0; no voters: 
I20240502 01:55:34.340098  7371 raft_consensus.cc:2781] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:34.340257  7371 raft_consensus.cc:482] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:34.340366  7371 raft_consensus.cc:3037] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:34.342433  7371 raft_consensus.cc:504] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:34.343122  7371 leader_election.cc:288] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 1 election: Requested vote from peers 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867), a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197)
I20240502 01:55:34.343673  6469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c58d6b2401c047cd9af56a66c2e00105" candidate_uuid: "218bc8c1f9ce46059898fc738179adf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07cdb213c77c4ffda1f999b66b725399"
I20240502 01:55:34.343744  6203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c58d6b2401c047cd9af56a66c2e00105" candidate_uuid: "218bc8c1f9ce46059898fc738179adf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c90ee440b44b708bccc2e2d5385cc0"
I20240502 01:55:34.343914  6469 raft_consensus.cc:3037] T c58d6b2401c047cd9af56a66c2e00105 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:34.343977  6203 raft_consensus.cc:3037] T c58d6b2401c047cd9af56a66c2e00105 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:34.346016  6469 raft_consensus.cc:2454] T c58d6b2401c047cd9af56a66c2e00105 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 218bc8c1f9ce46059898fc738179adf9 in term 1.
I20240502 01:55:34.346047  6203 raft_consensus.cc:2454] T c58d6b2401c047cd9af56a66c2e00105 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 218bc8c1f9ce46059898fc738179adf9 in term 1.
I20240502 01:55:34.346500  5998 leader_election.cc:302] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07cdb213c77c4ffda1f999b66b725399, 218bc8c1f9ce46059898fc738179adf9; no voters: 
I20240502 01:55:34.346844  7371 raft_consensus.cc:2781] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:34.347254  7371 raft_consensus.cc:686] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 [term 1 LEADER]: Becoming Leader. State: Replica: 218bc8c1f9ce46059898fc738179adf9, State: Running, Role: LEADER
I20240502 01:55:34.347776  7371 consensus_queue.cc:237] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:34.351375  5912 catalog_manager.cc:5484] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129). New cstate: current_term: 1 leader_uuid: "218bc8c1f9ce46059898fc738179adf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } }
I20240502 01:55:34.600740  7409 consensus_queue.cc:1035] T 74eadf4c939c4c6383732cd74c5ec9d6 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:34.608065  7416 consensus_queue.cc:1035] T 74eadf4c939c4c6383732cd74c5ec9d6 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:34.738380  7416 consensus_queue.cc:1035] T 55de74ae85e541a19008fd2fc2da3ffd P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:34.743322  7416 consensus_queue.cc:1035] T 55de74ae85e541a19008fd2fc2da3ffd P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:34.753963  7371 consensus_queue.cc:1035] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:34.777340  7447 consensus_queue.cc:1035] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:55:34.802685  7449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:34.803079  7449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:34.810884  7449 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'

INSERT report
    rows total: 2000
    time total: 217.599 ms
  time per row: 0.1088 ms
I20240502 01:55:35.182319  5920 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58156:
name: "rebalance_test_table_5"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:55:35.192625  6447 tablet_service.cc:1467] Processing CreateTablet for tablet e4d4a1bd80f44b668a2d504e47fb744a (DEFAULT_TABLE table=rebalance_test_table_5 [id=c02dbeef0b7241e78a12e4bd7ef79e47]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:35.193100  6447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4d4a1bd80f44b668a2d504e47fb744a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:35.194011  6047 tablet_service.cc:1467] Processing CreateTablet for tablet a398da35f18c4df394db82c5bf89aeb1 (DEFAULT_TABLE table=rebalance_test_table_5 [id=c02dbeef0b7241e78a12e4bd7ef79e47]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:35.194248  6181 tablet_service.cc:1467] Processing CreateTablet for tablet e4d4a1bd80f44b668a2d504e47fb744a (DEFAULT_TABLE table=rebalance_test_table_5 [id=c02dbeef0b7241e78a12e4bd7ef79e47]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:35.194482  6047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a398da35f18c4df394db82c5bf89aeb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:35.194409  6180 tablet_service.cc:1467] Processing CreateTablet for tablet d7922fd756344d14a336ff0c54afdf73 (DEFAULT_TABLE table=rebalance_test_table_5 [id=c02dbeef0b7241e78a12e4bd7ef79e47]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:35.194675  6181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4d4a1bd80f44b668a2d504e47fb744a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:35.194865  6180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7922fd756344d14a336ff0c54afdf73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:35.196328  6182 tablet_service.cc:1467] Processing CreateTablet for tablet a398da35f18c4df394db82c5bf89aeb1 (DEFAULT_TABLE table=rebalance_test_table_5 [id=c02dbeef0b7241e78a12e4bd7ef79e47]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:35.196795  6182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a398da35f18c4df394db82c5bf89aeb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:35.200728  6314 tablet_service.cc:1467] Processing CreateTablet for tablet e4d4a1bd80f44b668a2d504e47fb744a (DEFAULT_TABLE table=rebalance_test_table_5 [id=c02dbeef0b7241e78a12e4bd7ef79e47]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:35.201246  6314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4d4a1bd80f44b668a2d504e47fb744a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:35.204288  6315 tablet_service.cc:1467] Processing CreateTablet for tablet d7922fd756344d14a336ff0c54afdf73 (DEFAULT_TABLE table=rebalance_test_table_5 [id=c02dbeef0b7241e78a12e4bd7ef79e47]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:35.204721  6315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7922fd756344d14a336ff0c54afdf73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:35.207424  6316 tablet_service.cc:1467] Processing CreateTablet for tablet a398da35f18c4df394db82c5bf89aeb1 (DEFAULT_TABLE table=rebalance_test_table_5 [id=c02dbeef0b7241e78a12e4bd7ef79e47]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:35.207880  6316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a398da35f18c4df394db82c5bf89aeb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:35.210817  7514 tablet_bootstrap.cc:492] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0: Bootstrap starting.
I20240502 01:55:35.211642  6446 tablet_service.cc:1467] Processing CreateTablet for tablet d7922fd756344d14a336ff0c54afdf73 (DEFAULT_TABLE table=rebalance_test_table_5 [id=c02dbeef0b7241e78a12e4bd7ef79e47]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:35.212513  6446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7922fd756344d14a336ff0c54afdf73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:35.212972  7514 tablet_bootstrap.cc:654] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:35.221756  7514 tablet_bootstrap.cc:492] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0: No bootstrap required, opened a new log
I20240502 01:55:35.221977  7514 ts_tablet_manager.cc:1367] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0: Time spent bootstrapping tablet: real 0.011s	user 0.003s	sys 0.007s
I20240502 01:55:35.222990  7514 raft_consensus.cc:348] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:35.223361  7514 raft_consensus.cc:374] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:35.223515  7514 raft_consensus.cc:729] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c90ee440b44b708bccc2e2d5385cc0, State: Initialized, Role: FOLLOWER
I20240502 01:55:35.223886  7514 consensus_queue.cc:260] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:35.224603  7514 ts_tablet_manager.cc:1398] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 01:55:35.225029  7514 tablet_bootstrap.cc:492] T e4d4a1bd80f44b668a2d504e47fb744a P a2c90ee440b44b708bccc2e2d5385cc0: Bootstrap starting.
I20240502 01:55:35.229951  7516 tablet_bootstrap.cc:492] T d7922fd756344d14a336ff0c54afdf73 P 94ffd7aaf85b4e17881241b8396fa2f3: Bootstrap starting.
I20240502 01:55:35.230551  7517 tablet_bootstrap.cc:492] T d7922fd756344d14a336ff0c54afdf73 P 07cdb213c77c4ffda1f999b66b725399: Bootstrap starting.
I20240502 01:55:35.232231  7517 tablet_bootstrap.cc:654] T d7922fd756344d14a336ff0c54afdf73 P 07cdb213c77c4ffda1f999b66b725399: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:35.232583  7514 tablet_bootstrap.cc:654] T e4d4a1bd80f44b668a2d504e47fb744a P a2c90ee440b44b708bccc2e2d5385cc0: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:35.233891  7516 tablet_bootstrap.cc:654] T d7922fd756344d14a336ff0c54afdf73 P 94ffd7aaf85b4e17881241b8396fa2f3: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:35.234910  7514 tablet_bootstrap.cc:492] T e4d4a1bd80f44b668a2d504e47fb744a P a2c90ee440b44b708bccc2e2d5385cc0: No bootstrap required, opened a new log
I20240502 01:55:35.235119  7514 ts_tablet_manager.cc:1367] T e4d4a1bd80f44b668a2d504e47fb744a P a2c90ee440b44b708bccc2e2d5385cc0: Time spent bootstrapping tablet: real 0.010s	user 0.003s	sys 0.001s
I20240502 01:55:35.236001  7514 raft_consensus.cc:348] T e4d4a1bd80f44b668a2d504e47fb744a P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:35.236255  7514 raft_consensus.cc:374] T e4d4a1bd80f44b668a2d504e47fb744a P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:35.236291  7513 tablet_bootstrap.cc:492] T a398da35f18c4df394db82c5bf89aeb1 P 218bc8c1f9ce46059898fc738179adf9: Bootstrap starting.
I20240502 01:55:35.236371  7514 raft_consensus.cc:729] T e4d4a1bd80f44b668a2d504e47fb744a P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c90ee440b44b708bccc2e2d5385cc0, State: Initialized, Role: FOLLOWER
I20240502 01:55:35.236688  7514 consensus_queue.cc:260] T e4d4a1bd80f44b668a2d504e47fb744a P a2c90ee440b44b708bccc2e2d5385cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:35.237303  7516 tablet_bootstrap.cc:492] T d7922fd756344d14a336ff0c54afdf73 P 94ffd7aaf85b4e17881241b8396fa2f3: No bootstrap required, opened a new log
I20240502 01:55:35.237476  7516 ts_tablet_manager.cc:1367] T d7922fd756344d14a336ff0c54afdf73 P 94ffd7aaf85b4e17881241b8396fa2f3: Time spent bootstrapping tablet: real 0.008s	user 0.003s	sys 0.000s
I20240502 01:55:35.237993  7514 ts_tablet_manager.cc:1398] T e4d4a1bd80f44b668a2d504e47fb744a P a2c90ee440b44b708bccc2e2d5385cc0: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:55:35.238289  7514 tablet_bootstrap.cc:492] T a398da35f18c4df394db82c5bf89aeb1 P a2c90ee440b44b708bccc2e2d5385cc0: Bootstrap starting.
I20240502 01:55:35.239373  7513 tablet_bootstrap.cc:654] T a398da35f18c4df394db82c5bf89aeb1 P 218bc8c1f9ce46059898fc738179adf9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:35.239531  7514 tablet_bootstrap.cc:654] T a398da35f18c4df394db82c5bf89aeb1 P a2c90ee440b44b708bccc2e2d5385cc0: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:35.240677  7516 raft_consensus.cc:348] T d7922fd756344d14a336ff0c54afdf73 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:35.240926  7516 raft_consensus.cc:374] T d7922fd756344d14a336ff0c54afdf73 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:35.241016  7516 raft_consensus.cc:729] T d7922fd756344d14a336ff0c54afdf73 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94ffd7aaf85b4e17881241b8396fa2f3, State: Initialized, Role: FOLLOWER
I20240502 01:55:35.242244  7517 tablet_bootstrap.cc:492] T d7922fd756344d14a336ff0c54afdf73 P 07cdb213c77c4ffda1f999b66b725399: No bootstrap required, opened a new log
I20240502 01:55:35.242435  7517 ts_tablet_manager.cc:1367] T d7922fd756344d14a336ff0c54afdf73 P 07cdb213c77c4ffda1f999b66b725399: Time spent bootstrapping tablet: real 0.012s	user 0.003s	sys 0.000s
I20240502 01:55:35.242681  7516 consensus_queue.cc:260] T d7922fd756344d14a336ff0c54afdf73 P 94ffd7aaf85b4e17881241b8396fa2f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:35.243584  7513 tablet_bootstrap.cc:492] T a398da35f18c4df394db82c5bf89aeb1 P 218bc8c1f9ce46059898fc738179adf9: No bootstrap required, opened a new log
I20240502 01:55:35.244122  7517 raft_consensus.cc:348] T d7922fd756344d14a336ff0c54afdf73 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:35.244449  7517 raft_consensus.cc:374] T d7922fd756344d14a336ff0c54afdf73 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:35.244587  7517 raft_consensus.cc:729] T d7922fd756344d14a336ff0c54afdf73 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07cdb213c77c4ffda1f999b66b725399, State: Initialized, Role: FOLLOWER
I20240502 01:55:35.244938  7516 ts_tablet_manager.cc:1398] T d7922fd756344d14a336ff0c54afdf73 P 94ffd7aaf85b4e17881241b8396fa2f3: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.000s
I20240502 01:55:35.244906  7517 consensus_queue.cc:260] T d7922fd756344d14a336ff0c54afdf73 P 07cdb213c77c4ffda1f999b66b725399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:35.245160  7513 ts_tablet_manager.cc:1367] T a398da35f18c4df394db82c5bf89aeb1 P 218bc8c1f9ce46059898fc738179adf9: Time spent bootstrapping tablet: real 0.009s	user 0.006s	sys 0.000s
I20240502 01:55:35.246011  7513 raft_consensus.cc:348] T a398da35f18c4df394db82c5bf89aeb1 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:35.248152  7517 ts_tablet_manager.cc:1398] T d7922fd756344d14a336ff0c54afdf73 P 07cdb213c77c4ffda1f999b66b725399: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.000s
I20240502 01:55:35.248252  7516 tablet_bootstrap.cc:492] T e4d4a1bd80f44b668a2d504e47fb744a P 94ffd7aaf85b4e17881241b8396fa2f3: Bootstrap starting.
I20240502 01:55:35.248456  7514 tablet_bootstrap.cc:492] T a398da35f18c4df394db82c5bf89aeb1 P a2c90ee440b44b708bccc2e2d5385cc0: No bootstrap required, opened a new log
I20240502 01:55:35.248667  7514 ts_tablet_manager.cc:1367] T a398da35f18c4df394db82c5bf89aeb1 P a2c90ee440b44b708bccc2e2d5385cc0: Time spent bootstrapping tablet: real 0.010s	user 0.003s	sys 0.006s
I20240502 01:55:35.249399  7517 tablet_bootstrap.cc:492] T e4d4a1bd80f44b668a2d504e47fb744a P 07cdb213c77c4ffda1f999b66b725399: Bootstrap starting.
I20240502 01:55:35.249543  7514 raft_consensus.cc:348] T a398da35f18c4df394db82c5bf89aeb1 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:35.249729  7514 raft_consensus.cc:374] T a398da35f18c4df394db82c5bf89aeb1 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:35.249819  7514 raft_consensus.cc:729] T a398da35f18c4df394db82c5bf89aeb1 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c90ee440b44b708bccc2e2d5385cc0, State: Initialized, Role: FOLLOWER
I20240502 01:55:35.250054  7514 consensus_queue.cc:260] T a398da35f18c4df394db82c5bf89aeb1 P a2c90ee440b44b708bccc2e2d5385cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:35.250444  7513 raft_consensus.cc:374] T a398da35f18c4df394db82c5bf89aeb1 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:35.250617  7513 raft_consensus.cc:729] T a398da35f18c4df394db82c5bf89aeb1 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218bc8c1f9ce46059898fc738179adf9, State: Initialized, Role: FOLLOWER
I20240502 01:55:35.250896  7514 ts_tablet_manager.cc:1398] T a398da35f18c4df394db82c5bf89aeb1 P a2c90ee440b44b708bccc2e2d5385cc0: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240502 01:55:35.251367  7517 tablet_bootstrap.cc:654] T e4d4a1bd80f44b668a2d504e47fb744a P 07cdb213c77c4ffda1f999b66b725399: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:35.251279  7513 consensus_queue.cc:260] T a398da35f18c4df394db82c5bf89aeb1 P 218bc8c1f9ce46059898fc738179adf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:35.252421  7516 tablet_bootstrap.cc:654] T e4d4a1bd80f44b668a2d504e47fb744a P 94ffd7aaf85b4e17881241b8396fa2f3: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:35.254135  7447 raft_consensus.cc:482] T a398da35f18c4df394db82c5bf89aeb1 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:35.254127  7513 ts_tablet_manager.cc:1398] T a398da35f18c4df394db82c5bf89aeb1 P 218bc8c1f9ce46059898fc738179adf9: Time spent starting tablet: real 0.009s	user 0.002s	sys 0.000s
I20240502 01:55:35.254338  7447 raft_consensus.cc:504] T a398da35f18c4df394db82c5bf89aeb1 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:35.255525  7447 leader_election.cc:288] T a398da35f18c4df394db82c5bf89aeb1 P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197), 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245)
I20240502 01:55:35.256276  6204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a398da35f18c4df394db82c5bf89aeb1" candidate_uuid: "218bc8c1f9ce46059898fc738179adf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" is_pre_election: true
I20240502 01:55:35.256553  6204 raft_consensus.cc:2454] T a398da35f18c4df394db82c5bf89aeb1 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 218bc8c1f9ce46059898fc738179adf9 in term 0.
I20240502 01:55:35.257282  7516 tablet_bootstrap.cc:492] T e4d4a1bd80f44b668a2d504e47fb744a P 94ffd7aaf85b4e17881241b8396fa2f3: No bootstrap required, opened a new log
I20240502 01:55:35.257283  6001 leader_election.cc:302] T a398da35f18c4df394db82c5bf89aeb1 P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9, a2c90ee440b44b708bccc2e2d5385cc0; no voters: 
I20240502 01:55:35.257494  7516 ts_tablet_manager.cc:1367] T e4d4a1bd80f44b668a2d504e47fb744a P 94ffd7aaf85b4e17881241b8396fa2f3: Time spent bootstrapping tablet: real 0.009s	user 0.003s	sys 0.000s
I20240502 01:55:35.258762  7447 raft_consensus.cc:2781] T a398da35f18c4df394db82c5bf89aeb1 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:35.258646  7516 raft_consensus.cc:348] T e4d4a1bd80f44b668a2d504e47fb744a P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:35.258872  7447 raft_consensus.cc:482] T a398da35f18c4df394db82c5bf89aeb1 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:35.258913  7516 raft_consensus.cc:374] T e4d4a1bd80f44b668a2d504e47fb744a P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:35.258981  7447 raft_consensus.cc:3037] T a398da35f18c4df394db82c5bf89aeb1 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:35.258983  7516 raft_consensus.cc:729] T e4d4a1bd80f44b668a2d504e47fb744a P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94ffd7aaf85b4e17881241b8396fa2f3, State: Initialized, Role: FOLLOWER
I20240502 01:55:35.259259  7516 consensus_queue.cc:260] T e4d4a1bd80f44b668a2d504e47fb744a P 94ffd7aaf85b4e17881241b8396fa2f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:35.259914  7516 ts_tablet_manager.cc:1398] T e4d4a1bd80f44b668a2d504e47fb744a P 94ffd7aaf85b4e17881241b8396fa2f3: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 01:55:35.260252  7516 tablet_bootstrap.cc:492] T a398da35f18c4df394db82c5bf89aeb1 P 94ffd7aaf85b4e17881241b8396fa2f3: Bootstrap starting.
I20240502 01:55:35.262090  7516 tablet_bootstrap.cc:654] T a398da35f18c4df394db82c5bf89aeb1 P 94ffd7aaf85b4e17881241b8396fa2f3: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:35.263547  7447 raft_consensus.cc:504] T a398da35f18c4df394db82c5bf89aeb1 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:35.264590  7517 tablet_bootstrap.cc:492] T e4d4a1bd80f44b668a2d504e47fb744a P 07cdb213c77c4ffda1f999b66b725399: No bootstrap required, opened a new log
I20240502 01:55:35.264688  7447 leader_election.cc:288] T a398da35f18c4df394db82c5bf89aeb1 P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 1 election: Requested vote from peers a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197), 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245)
I20240502 01:55:35.264801  7517 ts_tablet_manager.cc:1367] T e4d4a1bd80f44b668a2d504e47fb744a P 07cdb213c77c4ffda1f999b66b725399: Time spent bootstrapping tablet: real 0.015s	user 0.011s	sys 0.003s
I20240502 01:55:35.265177  6204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a398da35f18c4df394db82c5bf89aeb1" candidate_uuid: "218bc8c1f9ce46059898fc738179adf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c90ee440b44b708bccc2e2d5385cc0"
I20240502 01:55:35.265401  6204 raft_consensus.cc:3037] T a398da35f18c4df394db82c5bf89aeb1 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:35.265797  7517 raft_consensus.cc:348] T e4d4a1bd80f44b668a2d504e47fb744a P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:35.266072  7517 raft_consensus.cc:374] T e4d4a1bd80f44b668a2d504e47fb744a P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:35.266155  7517 raft_consensus.cc:729] T e4d4a1bd80f44b668a2d504e47fb744a P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07cdb213c77c4ffda1f999b66b725399, State: Initialized, Role: FOLLOWER
I20240502 01:55:35.266500  7517 consensus_queue.cc:260] T e4d4a1bd80f44b668a2d504e47fb744a P 07cdb213c77c4ffda1f999b66b725399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:35.267437  6204 raft_consensus.cc:2454] T a398da35f18c4df394db82c5bf89aeb1 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 218bc8c1f9ce46059898fc738179adf9 in term 1.
I20240502 01:55:35.268057  7517 ts_tablet_manager.cc:1398] T e4d4a1bd80f44b668a2d504e47fb744a P 07cdb213c77c4ffda1f999b66b725399: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:55:35.268354  6001 leader_election.cc:302] T a398da35f18c4df394db82c5bf89aeb1 P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9, a2c90ee440b44b708bccc2e2d5385cc0; no voters: 
I20240502 01:55:35.268693  7447 raft_consensus.cc:2781] T a398da35f18c4df394db82c5bf89aeb1 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:35.268946  7447 raft_consensus.cc:686] T a398da35f18c4df394db82c5bf89aeb1 P 218bc8c1f9ce46059898fc738179adf9 [term 1 LEADER]: Becoming Leader. State: Replica: 218bc8c1f9ce46059898fc738179adf9, State: Running, Role: LEADER
I20240502 01:55:35.269284  7447 consensus_queue.cc:237] T a398da35f18c4df394db82c5bf89aeb1 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:35.270072  7516 tablet_bootstrap.cc:492] T a398da35f18c4df394db82c5bf89aeb1 P 94ffd7aaf85b4e17881241b8396fa2f3: No bootstrap required, opened a new log
I20240502 01:55:35.270238  7516 ts_tablet_manager.cc:1367] T a398da35f18c4df394db82c5bf89aeb1 P 94ffd7aaf85b4e17881241b8396fa2f3: Time spent bootstrapping tablet: real 0.010s	user 0.003s	sys 0.000s
I20240502 01:55:35.271135  7516 raft_consensus.cc:348] T a398da35f18c4df394db82c5bf89aeb1 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:35.271365  7516 raft_consensus.cc:374] T a398da35f18c4df394db82c5bf89aeb1 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:35.271471  7516 raft_consensus.cc:729] T a398da35f18c4df394db82c5bf89aeb1 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94ffd7aaf85b4e17881241b8396fa2f3, State: Initialized, Role: FOLLOWER
I20240502 01:55:35.271764  7516 consensus_queue.cc:260] T a398da35f18c4df394db82c5bf89aeb1 P 94ffd7aaf85b4e17881241b8396fa2f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:35.273344  7516 ts_tablet_manager.cc:1398] T a398da35f18c4df394db82c5bf89aeb1 P 94ffd7aaf85b4e17881241b8396fa2f3: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 01:55:35.275985  5920 catalog_manager.cc:5484] T a398da35f18c4df394db82c5bf89aeb1 P 218bc8c1f9ce46059898fc738179adf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129). New cstate: current_term: 1 leader_uuid: "218bc8c1f9ce46059898fc738179adf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } }
I20240502 01:55:35.281723  6335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a398da35f18c4df394db82c5bf89aeb1" candidate_uuid: "218bc8c1f9ce46059898fc738179adf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" is_pre_election: true
I20240502 01:55:35.281845  6336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a398da35f18c4df394db82c5bf89aeb1" candidate_uuid: "218bc8c1f9ce46059898fc738179adf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ffd7aaf85b4e17881241b8396fa2f3"
I20240502 01:55:35.282019  6335 raft_consensus.cc:2454] T a398da35f18c4df394db82c5bf89aeb1 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 218bc8c1f9ce46059898fc738179adf9 in term 0.
I20240502 01:55:35.401985  7317 raft_consensus.cc:482] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:35.402204  7317 raft_consensus.cc:504] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:35.403005  7317 leader_election.cc:288] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245), 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867)
I20240502 01:55:35.403699  6469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7922fd756344d14a336ff0c54afdf73" candidate_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07cdb213c77c4ffda1f999b66b725399" is_pre_election: true
I20240502 01:55:35.404021  6469 raft_consensus.cc:2454] T d7922fd756344d14a336ff0c54afdf73 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c90ee440b44b708bccc2e2d5385cc0 in term 0.
I20240502 01:55:35.404506  6134 leader_election.cc:302] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07cdb213c77c4ffda1f999b66b725399, a2c90ee440b44b708bccc2e2d5385cc0; no voters: 
I20240502 01:55:35.404839  7317 raft_consensus.cc:2781] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:35.404942  7317 raft_consensus.cc:482] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:35.405067  7317 raft_consensus.cc:3037] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:35.407146  7317 raft_consensus.cc:504] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:35.407895  7317 leader_election.cc:288] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 [CANDIDATE]: Term 1 election: Requested vote from peers 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245), 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867)
I20240502 01:55:35.408490  6469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7922fd756344d14a336ff0c54afdf73" candidate_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07cdb213c77c4ffda1f999b66b725399"
I20240502 01:55:35.408721  6469 raft_consensus.cc:3037] T d7922fd756344d14a336ff0c54afdf73 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:35.410195  6335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7922fd756344d14a336ff0c54afdf73" candidate_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ffd7aaf85b4e17881241b8396fa2f3"
I20240502 01:55:35.410140  6336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7922fd756344d14a336ff0c54afdf73" candidate_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" is_pre_election: true
I20240502 01:55:35.410436  6335 raft_consensus.cc:3037] T d7922fd756344d14a336ff0c54afdf73 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:35.411327  6469 raft_consensus.cc:2454] T d7922fd756344d14a336ff0c54afdf73 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c90ee440b44b708bccc2e2d5385cc0 in term 1.
I20240502 01:55:35.411744  6134 leader_election.cc:302] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07cdb213c77c4ffda1f999b66b725399, a2c90ee440b44b708bccc2e2d5385cc0; no voters: 
I20240502 01:55:35.412061  7317 raft_consensus.cc:2781] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:35.412606  7317 raft_consensus.cc:686] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 LEADER]: Becoming Leader. State: Replica: a2c90ee440b44b708bccc2e2d5385cc0, State: Running, Role: LEADER
I20240502 01:55:35.412925  6335 raft_consensus.cc:2454] T d7922fd756344d14a336ff0c54afdf73 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c90ee440b44b708bccc2e2d5385cc0 in term 1.
I20240502 01:55:35.413316  7317 consensus_queue.cc:237] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:35.416692  5916 catalog_manager.cc:5484] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130). New cstate: current_term: 1 leader_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } }
I20240502 01:55:35.457290  7317 raft_consensus.cc:482] T e4d4a1bd80f44b668a2d504e47fb744a P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:35.457495  7317 raft_consensus.cc:504] T e4d4a1bd80f44b668a2d504e47fb744a P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:35.458235  7317 leader_election.cc:288] T e4d4a1bd80f44b668a2d504e47fb744a P a2c90ee440b44b708bccc2e2d5385cc0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245), 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867)
I20240502 01:55:35.458794  6335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4d4a1bd80f44b668a2d504e47fb744a" candidate_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" is_pre_election: true
I20240502 01:55:35.459055  6335 raft_consensus.cc:2454] T e4d4a1bd80f44b668a2d504e47fb744a P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c90ee440b44b708bccc2e2d5385cc0 in term 0.
I20240502 01:55:35.459470  6469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4d4a1bd80f44b668a2d504e47fb744a" candidate_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07cdb213c77c4ffda1f999b66b725399" is_pre_election: true
I20240502 01:55:35.459712  6469 raft_consensus.cc:2454] T e4d4a1bd80f44b668a2d504e47fb744a P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c90ee440b44b708bccc2e2d5385cc0 in term 0.
I20240502 01:55:35.459614  6136 leader_election.cc:302] T e4d4a1bd80f44b668a2d504e47fb744a P a2c90ee440b44b708bccc2e2d5385cc0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94ffd7aaf85b4e17881241b8396fa2f3, a2c90ee440b44b708bccc2e2d5385cc0; no voters: 
I20240502 01:55:35.460098  7317 raft_consensus.cc:2781] T e4d4a1bd80f44b668a2d504e47fb744a P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:35.460261  7317 raft_consensus.cc:482] T e4d4a1bd80f44b668a2d504e47fb744a P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:35.460366  7317 raft_consensus.cc:3037] T e4d4a1bd80f44b668a2d504e47fb744a P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:35.482017  7317 raft_consensus.cc:504] T e4d4a1bd80f44b668a2d504e47fb744a P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:35.483430  7317 leader_election.cc:288] T e4d4a1bd80f44b668a2d504e47fb744a P a2c90ee440b44b708bccc2e2d5385cc0 [CANDIDATE]: Term 1 election: Requested vote from peers 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245), 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867)
I20240502 01:55:35.483403  6335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4d4a1bd80f44b668a2d504e47fb744a" candidate_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ffd7aaf85b4e17881241b8396fa2f3"
I20240502 01:55:35.483425  6469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4d4a1bd80f44b668a2d504e47fb744a" candidate_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07cdb213c77c4ffda1f999b66b725399"
I20240502 01:55:35.483654  6335 raft_consensus.cc:3037] T e4d4a1bd80f44b668a2d504e47fb744a P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:35.483666  6469 raft_consensus.cc:3037] T e4d4a1bd80f44b668a2d504e47fb744a P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:35.486593  6335 raft_consensus.cc:2454] T e4d4a1bd80f44b668a2d504e47fb744a P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c90ee440b44b708bccc2e2d5385cc0 in term 1.
I20240502 01:55:35.486653  6469 raft_consensus.cc:2454] T e4d4a1bd80f44b668a2d504e47fb744a P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c90ee440b44b708bccc2e2d5385cc0 in term 1.
I20240502 01:55:35.487236  6136 leader_election.cc:302] T e4d4a1bd80f44b668a2d504e47fb744a P a2c90ee440b44b708bccc2e2d5385cc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94ffd7aaf85b4e17881241b8396fa2f3, a2c90ee440b44b708bccc2e2d5385cc0; no voters: 
I20240502 01:55:35.487710  7317 raft_consensus.cc:2781] T e4d4a1bd80f44b668a2d504e47fb744a P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:35.488023  7317 raft_consensus.cc:686] T e4d4a1bd80f44b668a2d504e47fb744a P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 LEADER]: Becoming Leader. State: Replica: a2c90ee440b44b708bccc2e2d5385cc0, State: Running, Role: LEADER
I20240502 01:55:35.488366  7317 consensus_queue.cc:237] T e4d4a1bd80f44b668a2d504e47fb744a P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:35.494316  5916 catalog_manager.cc:5484] T e4d4a1bd80f44b668a2d504e47fb744a P a2c90ee440b44b708bccc2e2d5385cc0 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130). New cstate: current_term: 1 leader_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } }
I20240502 01:55:35.773797  7447 consensus_queue.cc:1035] T a398da35f18c4df394db82c5bf89aeb1 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:35.778453  6335 raft_consensus.cc:3037] T a398da35f18c4df394db82c5bf89aeb1 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:35.781695  7534 consensus_queue.cc:1035] T a398da35f18c4df394db82c5bf89aeb1 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:35.852159  7317 consensus_queue.cc:1035] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:55:35.891206  7532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:35.891554  7532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 01:55:35.895417  7529 consensus_queue.cc:1035] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:55:35.901824  7532 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240502 01:55:35.959039  7529 consensus_queue.cc:1035] T e4d4a1bd80f44b668a2d504e47fb744a P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:35.981806  7529 consensus_queue.cc:1035] T e4d4a1bd80f44b668a2d504e47fb744a P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 291.34 ms
  time per row: 0.14567 ms
I20240502 01:55:36.340720  5922 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58156:
name: "rebalance_test_table_6"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:55:36.350253  6182 tablet_service.cc:1467] Processing CreateTablet for tablet 4486b413e5c14ec6953e4895ee5797cf (DEFAULT_TABLE table=rebalance_test_table_6 [id=5606c3da105b4b86868c6c51ecdd39aa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:36.350364  6446 tablet_service.cc:1467] Processing CreateTablet for tablet 4486b413e5c14ec6953e4895ee5797cf (DEFAULT_TABLE table=rebalance_test_table_6 [id=5606c3da105b4b86868c6c51ecdd39aa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:36.350787  6446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4486b413e5c14ec6953e4895ee5797cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:36.350808  6182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4486b413e5c14ec6953e4895ee5797cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:36.352345  6047 tablet_service.cc:1467] Processing CreateTablet for tablet 4486b413e5c14ec6953e4895ee5797cf (DEFAULT_TABLE table=rebalance_test_table_6 [id=5606c3da105b4b86868c6c51ecdd39aa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:36.352514  6046 tablet_service.cc:1467] Processing CreateTablet for tablet 73ed91df7ff04846991c95b30a86ea7e (DEFAULT_TABLE table=rebalance_test_table_6 [id=5606c3da105b4b86868c6c51ecdd39aa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:36.352834  6047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4486b413e5c14ec6953e4895ee5797cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:36.352994  6448 tablet_service.cc:1467] Processing CreateTablet for tablet 73ed91df7ff04846991c95b30a86ea7e (DEFAULT_TABLE table=rebalance_test_table_6 [id=5606c3da105b4b86868c6c51ecdd39aa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:36.353375  6448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73ed91df7ff04846991c95b30a86ea7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:36.353348  6315 tablet_service.cc:1467] Processing CreateTablet for tablet 899963a8260a42c4b73fb80eda2ecaa9 (DEFAULT_TABLE table=rebalance_test_table_6 [id=5606c3da105b4b86868c6c51ecdd39aa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:36.353783  6315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 899963a8260a42c4b73fb80eda2ecaa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:36.354882  6180 tablet_service.cc:1467] Processing CreateTablet for tablet 73ed91df7ff04846991c95b30a86ea7e (DEFAULT_TABLE table=rebalance_test_table_6 [id=5606c3da105b4b86868c6c51ecdd39aa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:36.355317  6180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73ed91df7ff04846991c95b30a86ea7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:36.356087  6048 tablet_service.cc:1467] Processing CreateTablet for tablet 899963a8260a42c4b73fb80eda2ecaa9 (DEFAULT_TABLE table=rebalance_test_table_6 [id=5606c3da105b4b86868c6c51ecdd39aa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:36.356520  6048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 899963a8260a42c4b73fb80eda2ecaa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:36.358423  6046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73ed91df7ff04846991c95b30a86ea7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:36.352612  6447 tablet_service.cc:1467] Processing CreateTablet for tablet 899963a8260a42c4b73fb80eda2ecaa9 (DEFAULT_TABLE table=rebalance_test_table_6 [id=5606c3da105b4b86868c6c51ecdd39aa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:36.364054  6447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 899963a8260a42c4b73fb80eda2ecaa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:36.370447  7596 tablet_bootstrap.cc:492] T 4486b413e5c14ec6953e4895ee5797cf P 218bc8c1f9ce46059898fc738179adf9: Bootstrap starting.
I20240502 01:55:36.372510  7596 tablet_bootstrap.cc:654] T 4486b413e5c14ec6953e4895ee5797cf P 218bc8c1f9ce46059898fc738179adf9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:36.374967  7597 tablet_bootstrap.cc:492] T 899963a8260a42c4b73fb80eda2ecaa9 P 94ffd7aaf85b4e17881241b8396fa2f3: Bootstrap starting.
I20240502 01:55:36.377736  7597 tablet_bootstrap.cc:654] T 899963a8260a42c4b73fb80eda2ecaa9 P 94ffd7aaf85b4e17881241b8396fa2f3: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:36.380556  7596 tablet_bootstrap.cc:492] T 4486b413e5c14ec6953e4895ee5797cf P 218bc8c1f9ce46059898fc738179adf9: No bootstrap required, opened a new log
I20240502 01:55:36.380818  7596 ts_tablet_manager.cc:1367] T 4486b413e5c14ec6953e4895ee5797cf P 218bc8c1f9ce46059898fc738179adf9: Time spent bootstrapping tablet: real 0.010s	user 0.004s	sys 0.006s
I20240502 01:55:36.381937  7596 raft_consensus.cc:348] T 4486b413e5c14ec6953e4895ee5797cf P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 01:55:36.382308  7596 raft_consensus.cc:374] T 4486b413e5c14ec6953e4895ee5797cf P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:36.382452  7596 raft_consensus.cc:729] T 4486b413e5c14ec6953e4895ee5797cf P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218bc8c1f9ce46059898fc738179adf9, State: Initialized, Role: FOLLOWER
I20240502 01:55:36.382787  7596 consensus_queue.cc:260] T 4486b413e5c14ec6953e4895ee5797cf P 218bc8c1f9ce46059898fc738179adf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 01:55:36.384037  7596 ts_tablet_manager.cc:1398] T 4486b413e5c14ec6953e4895ee5797cf P 218bc8c1f9ce46059898fc738179adf9: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 01:55:36.384486  7596 tablet_bootstrap.cc:492] T 73ed91df7ff04846991c95b30a86ea7e P 218bc8c1f9ce46059898fc738179adf9: Bootstrap starting.
I20240502 01:55:36.386336  7596 tablet_bootstrap.cc:654] T 73ed91df7ff04846991c95b30a86ea7e P 218bc8c1f9ce46059898fc738179adf9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:36.393713  7601 tablet_bootstrap.cc:492] T 899963a8260a42c4b73fb80eda2ecaa9 P 07cdb213c77c4ffda1f999b66b725399: Bootstrap starting.
I20240502 01:55:36.394637  7595 tablet_bootstrap.cc:492] T 4486b413e5c14ec6953e4895ee5797cf P a2c90ee440b44b708bccc2e2d5385cc0: Bootstrap starting.
I20240502 01:55:36.395531  7596 tablet_bootstrap.cc:492] T 73ed91df7ff04846991c95b30a86ea7e P 218bc8c1f9ce46059898fc738179adf9: No bootstrap required, opened a new log
I20240502 01:55:36.395599  7597 tablet_bootstrap.cc:492] T 899963a8260a42c4b73fb80eda2ecaa9 P 94ffd7aaf85b4e17881241b8396fa2f3: No bootstrap required, opened a new log
I20240502 01:55:36.395921  7597 ts_tablet_manager.cc:1367] T 899963a8260a42c4b73fb80eda2ecaa9 P 94ffd7aaf85b4e17881241b8396fa2f3: Time spent bootstrapping tablet: real 0.021s	user 0.004s	sys 0.011s
I20240502 01:55:36.396660  7601 tablet_bootstrap.cc:654] T 899963a8260a42c4b73fb80eda2ecaa9 P 07cdb213c77c4ffda1f999b66b725399: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:36.396853  7597 raft_consensus.cc:348] T 899963a8260a42c4b73fb80eda2ecaa9 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:36.397208  7597 raft_consensus.cc:374] T 899963a8260a42c4b73fb80eda2ecaa9 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:36.397354  7597 raft_consensus.cc:729] T 899963a8260a42c4b73fb80eda2ecaa9 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94ffd7aaf85b4e17881241b8396fa2f3, State: Initialized, Role: FOLLOWER
I20240502 01:55:36.395741  7596 ts_tablet_manager.cc:1367] T 73ed91df7ff04846991c95b30a86ea7e P 218bc8c1f9ce46059898fc738179adf9: Time spent bootstrapping tablet: real 0.011s	user 0.009s	sys 0.000s
I20240502 01:55:36.397554  7595 tablet_bootstrap.cc:654] T 4486b413e5c14ec6953e4895ee5797cf P a2c90ee440b44b708bccc2e2d5385cc0: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:36.397789  7597 consensus_queue.cc:260] T 899963a8260a42c4b73fb80eda2ecaa9 P 94ffd7aaf85b4e17881241b8396fa2f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:36.398548  7596 raft_consensus.cc:348] T 73ed91df7ff04846991c95b30a86ea7e P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:36.398870  7596 raft_consensus.cc:374] T 73ed91df7ff04846991c95b30a86ea7e P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:36.399012  7597 ts_tablet_manager.cc:1398] T 899963a8260a42c4b73fb80eda2ecaa9 P 94ffd7aaf85b4e17881241b8396fa2f3: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:55:36.398988  7596 raft_consensus.cc:729] T 73ed91df7ff04846991c95b30a86ea7e P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218bc8c1f9ce46059898fc738179adf9, State: Initialized, Role: FOLLOWER
I20240502 01:55:36.399314  7596 consensus_queue.cc:260] T 73ed91df7ff04846991c95b30a86ea7e P 218bc8c1f9ce46059898fc738179adf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:36.400740  7596 ts_tablet_manager.cc:1398] T 73ed91df7ff04846991c95b30a86ea7e P 218bc8c1f9ce46059898fc738179adf9: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
I20240502 01:55:36.400996  7601 tablet_bootstrap.cc:492] T 899963a8260a42c4b73fb80eda2ecaa9 P 07cdb213c77c4ffda1f999b66b725399: No bootstrap required, opened a new log
I20240502 01:55:36.401176  7601 ts_tablet_manager.cc:1367] T 899963a8260a42c4b73fb80eda2ecaa9 P 07cdb213c77c4ffda1f999b66b725399: Time spent bootstrapping tablet: real 0.008s	user 0.000s	sys 0.006s
I20240502 01:55:36.401983  7595 tablet_bootstrap.cc:492] T 4486b413e5c14ec6953e4895ee5797cf P a2c90ee440b44b708bccc2e2d5385cc0: No bootstrap required, opened a new log
I20240502 01:55:36.402169  7595 ts_tablet_manager.cc:1367] T 4486b413e5c14ec6953e4895ee5797cf P a2c90ee440b44b708bccc2e2d5385cc0: Time spent bootstrapping tablet: real 0.008s	user 0.003s	sys 0.001s
I20240502 01:55:36.403046  7595 raft_consensus.cc:348] T 4486b413e5c14ec6953e4895ee5797cf P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 01:55:36.403281  7595 raft_consensus.cc:374] T 4486b413e5c14ec6953e4895ee5797cf P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:36.403381  7595 raft_consensus.cc:729] T 4486b413e5c14ec6953e4895ee5797cf P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c90ee440b44b708bccc2e2d5385cc0, State: Initialized, Role: FOLLOWER
I20240502 01:55:36.404003  7596 tablet_bootstrap.cc:492] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9: Bootstrap starting.
I20240502 01:55:36.404454  7601 raft_consensus.cc:348] T 899963a8260a42c4b73fb80eda2ecaa9 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:36.405797  7601 raft_consensus.cc:374] T 899963a8260a42c4b73fb80eda2ecaa9 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:36.405867  7596 tablet_bootstrap.cc:654] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:36.405920  7601 raft_consensus.cc:729] T 899963a8260a42c4b73fb80eda2ecaa9 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07cdb213c77c4ffda1f999b66b725399, State: Initialized, Role: FOLLOWER
I20240502 01:55:36.406242  7595 consensus_queue.cc:260] T 4486b413e5c14ec6953e4895ee5797cf P a2c90ee440b44b708bccc2e2d5385cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 01:55:36.406977  7595 ts_tablet_manager.cc:1398] T 4486b413e5c14ec6953e4895ee5797cf P a2c90ee440b44b708bccc2e2d5385cc0: Time spent starting tablet: real 0.005s	user 0.000s	sys 0.002s
I20240502 01:55:36.407333  7595 tablet_bootstrap.cc:492] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0: Bootstrap starting.
I20240502 01:55:36.408085  7601 consensus_queue.cc:260] T 899963a8260a42c4b73fb80eda2ecaa9 P 07cdb213c77c4ffda1f999b66b725399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:36.408862  7596 tablet_bootstrap.cc:492] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9: No bootstrap required, opened a new log
I20240502 01:55:36.409044  7595 tablet_bootstrap.cc:654] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:36.409111  7596 ts_tablet_manager.cc:1367] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.000s
I20240502 01:55:36.409941  7596 raft_consensus.cc:348] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:36.410183  7596 raft_consensus.cc:374] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:36.410322  7596 raft_consensus.cc:729] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218bc8c1f9ce46059898fc738179adf9, State: Initialized, Role: FOLLOWER
I20240502 01:55:36.410694  7596 consensus_queue.cc:260] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:36.414072  7596 ts_tablet_manager.cc:1398] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9: Time spent starting tablet: real 0.005s	user 0.001s	sys 0.001s
I20240502 01:55:36.415591  7601 ts_tablet_manager.cc:1398] T 899963a8260a42c4b73fb80eda2ecaa9 P 07cdb213c77c4ffda1f999b66b725399: Time spent starting tablet: real 0.012s	user 0.002s	sys 0.000s
I20240502 01:55:36.416175  7601 tablet_bootstrap.cc:492] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399: Bootstrap starting.
I20240502 01:55:36.418063  7601 tablet_bootstrap.cc:654] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:36.418411  7534 raft_consensus.cc:482] T 73ed91df7ff04846991c95b30a86ea7e P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:36.418603  7534 raft_consensus.cc:504] T 73ed91df7ff04846991c95b30a86ea7e P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:36.419417  7534 leader_election.cc:288] T 73ed91df7ff04846991c95b30a86ea7e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197), 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867)
I20240502 01:55:36.419950  6204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ed91df7ff04846991c95b30a86ea7e" candidate_uuid: "218bc8c1f9ce46059898fc738179adf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" is_pre_election: true
I20240502 01:55:36.420183  6467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ed91df7ff04846991c95b30a86ea7e" candidate_uuid: "218bc8c1f9ce46059898fc738179adf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07cdb213c77c4ffda1f999b66b725399" is_pre_election: true
W20240502 01:55:36.420890  6001 leader_election.cc:341] T 73ed91df7ff04846991c95b30a86ea7e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197): Illegal state: must be running to vote when last-logged opid is not known
W20240502 01:55:36.421291  5998 leader_election.cc:341] T 73ed91df7ff04846991c95b30a86ea7e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:55:36.421416  5998 leader_election.cc:302] T 73ed91df7ff04846991c95b30a86ea7e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9; no voters: 07cdb213c77c4ffda1f999b66b725399, a2c90ee440b44b708bccc2e2d5385cc0
I20240502 01:55:36.421695  7534 raft_consensus.cc:2726] T 73ed91df7ff04846991c95b30a86ea7e P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:55:36.422643  7601 tablet_bootstrap.cc:492] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399: No bootstrap required, opened a new log
I20240502 01:55:36.422842  7601 ts_tablet_manager.cc:1367] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399: Time spent bootstrapping tablet: real 0.007s	user 0.003s	sys 0.000s
I20240502 01:55:36.423714  7601 raft_consensus.cc:348] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:36.424008  7601 raft_consensus.cc:374] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:36.424093  7601 raft_consensus.cc:729] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07cdb213c77c4ffda1f999b66b725399, State: Initialized, Role: FOLLOWER
I20240502 01:55:36.424337  7601 consensus_queue.cc:260] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:36.424844  7601 ts_tablet_manager.cc:1398] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:55:36.425240  7601 tablet_bootstrap.cc:492] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399: Bootstrap starting.
I20240502 01:55:36.425344  7595 tablet_bootstrap.cc:492] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0: No bootstrap required, opened a new log
I20240502 01:55:36.425519  7595 ts_tablet_manager.cc:1367] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0: Time spent bootstrapping tablet: real 0.018s	user 0.011s	sys 0.004s
I20240502 01:55:36.426219  7534 raft_consensus.cc:482] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:36.426326  7595 raft_consensus.cc:348] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:36.426409  7534 raft_consensus.cc:504] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:36.426548  7595 raft_consensus.cc:374] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:36.426610  7595 raft_consensus.cc:729] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c90ee440b44b708bccc2e2d5385cc0, State: Initialized, Role: FOLLOWER
I20240502 01:55:36.426857  7595 consensus_queue.cc:260] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:36.427098  7534 leader_election.cc:288] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867), 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245)
I20240502 01:55:36.427564  7595 ts_tablet_manager.cc:1398] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
I20240502 01:55:36.427997  6334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "899963a8260a42c4b73fb80eda2ecaa9" candidate_uuid: "218bc8c1f9ce46059898fc738179adf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" is_pre_election: true
I20240502 01:55:36.428247  6334 raft_consensus.cc:2454] T 899963a8260a42c4b73fb80eda2ecaa9 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 218bc8c1f9ce46059898fc738179adf9 in term 0.
I20240502 01:55:36.428740  6000 leader_election.cc:302] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9, 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 
I20240502 01:55:36.429011  7534 raft_consensus.cc:2781] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:36.429105  7534 raft_consensus.cc:482] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:36.429023  6467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "899963a8260a42c4b73fb80eda2ecaa9" candidate_uuid: "218bc8c1f9ce46059898fc738179adf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07cdb213c77c4ffda1f999b66b725399" is_pre_election: true
I20240502 01:55:36.429194  7534 raft_consensus.cc:3037] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:36.429240  6467 raft_consensus.cc:2454] T 899963a8260a42c4b73fb80eda2ecaa9 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 218bc8c1f9ce46059898fc738179adf9 in term 0.
I20240502 01:55:36.430294  7601 tablet_bootstrap.cc:654] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:36.431915  7534 raft_consensus.cc:504] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:36.432929  7601 tablet_bootstrap.cc:492] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399: No bootstrap required, opened a new log
I20240502 01:55:36.433110  7601 ts_tablet_manager.cc:1367] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399: Time spent bootstrapping tablet: real 0.008s	user 0.003s	sys 0.000s
I20240502 01:55:36.433223  6467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "899963a8260a42c4b73fb80eda2ecaa9" candidate_uuid: "218bc8c1f9ce46059898fc738179adf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07cdb213c77c4ffda1f999b66b725399"
I20240502 01:55:36.433455  6467 raft_consensus.cc:3037] T 899963a8260a42c4b73fb80eda2ecaa9 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:36.433579  6334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "899963a8260a42c4b73fb80eda2ecaa9" candidate_uuid: "218bc8c1f9ce46059898fc738179adf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ffd7aaf85b4e17881241b8396fa2f3"
I20240502 01:55:36.433794  6334 raft_consensus.cc:3037] T 899963a8260a42c4b73fb80eda2ecaa9 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:36.434139  7601 raft_consensus.cc:348] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 01:55:36.434378  7601 raft_consensus.cc:374] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:36.434588  7601 raft_consensus.cc:729] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07cdb213c77c4ffda1f999b66b725399, State: Initialized, Role: FOLLOWER
I20240502 01:55:36.434863  7601 consensus_queue.cc:260] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 01:55:36.435410  7601 ts_tablet_manager.cc:1398] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:55:36.436336  6467 raft_consensus.cc:2454] T 899963a8260a42c4b73fb80eda2ecaa9 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 218bc8c1f9ce46059898fc738179adf9 in term 1.
I20240502 01:55:36.436496  6334 raft_consensus.cc:2454] T 899963a8260a42c4b73fb80eda2ecaa9 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 218bc8c1f9ce46059898fc738179adf9 in term 1.
I20240502 01:55:36.436879  5998 leader_election.cc:302] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07cdb213c77c4ffda1f999b66b725399, 218bc8c1f9ce46059898fc738179adf9; no voters: 
I20240502 01:55:36.436926  7534 leader_election.cc:288] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 1 election: Requested vote from peers 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867), 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245)
I20240502 01:55:36.437189  7534 raft_consensus.cc:2781] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:36.437356  7534 raft_consensus.cc:686] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 [term 1 LEADER]: Becoming Leader. State: Replica: 218bc8c1f9ce46059898fc738179adf9, State: Running, Role: LEADER
I20240502 01:55:36.437613  7534 consensus_queue.cc:237] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:36.440505  5922 catalog_manager.cc:5484] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129). New cstate: current_term: 1 leader_uuid: "218bc8c1f9ce46059898fc738179adf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } }
I20240502 01:55:36.559393  7254 raft_consensus.cc:482] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:36.559588  7254 raft_consensus.cc:504] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:36.560369  7254 leader_election.cc:288] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227), a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197)
I20240502 01:55:36.560994  6068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ed91df7ff04846991c95b30a86ea7e" candidate_uuid: "07cdb213c77c4ffda1f999b66b725399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218bc8c1f9ce46059898fc738179adf9" is_pre_election: true
I20240502 01:55:36.560993  6204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ed91df7ff04846991c95b30a86ea7e" candidate_uuid: "07cdb213c77c4ffda1f999b66b725399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" is_pre_election: true
I20240502 01:55:36.561249  6068 raft_consensus.cc:2454] T 73ed91df7ff04846991c95b30a86ea7e P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07cdb213c77c4ffda1f999b66b725399 in term 0.
I20240502 01:55:36.561252  6204 raft_consensus.cc:2454] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07cdb213c77c4ffda1f999b66b725399 in term 0.
I20240502 01:55:36.561726  6403 leader_election.cc:302] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 07cdb213c77c4ffda1f999b66b725399, 218bc8c1f9ce46059898fc738179adf9, a2c90ee440b44b708bccc2e2d5385cc0; no voters: 
I20240502 01:55:36.562038  7254 raft_consensus.cc:2781] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:36.562186  7254 raft_consensus.cc:482] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:36.562294  7254 raft_consensus.cc:3037] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:36.562402  7609 raft_consensus.cc:482] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:36.562552  7609 raft_consensus.cc:504] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 01:55:36.563320  7609 leader_election.cc:288] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197), 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227)
I20240502 01:55:36.563841  6204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4486b413e5c14ec6953e4895ee5797cf" candidate_uuid: "07cdb213c77c4ffda1f999b66b725399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" is_pre_election: true
I20240502 01:55:36.563889  6068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4486b413e5c14ec6953e4895ee5797cf" candidate_uuid: "07cdb213c77c4ffda1f999b66b725399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218bc8c1f9ce46059898fc738179adf9" is_pre_election: true
I20240502 01:55:36.564152  6068 raft_consensus.cc:2454] T 4486b413e5c14ec6953e4895ee5797cf P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07cdb213c77c4ffda1f999b66b725399 in term 0.
I20240502 01:55:36.564139  6204 raft_consensus.cc:2454] T 4486b413e5c14ec6953e4895ee5797cf P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07cdb213c77c4ffda1f999b66b725399 in term 0.
I20240502 01:55:36.564576  6403 leader_election.cc:302] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07cdb213c77c4ffda1f999b66b725399, a2c90ee440b44b708bccc2e2d5385cc0; no voters: 
I20240502 01:55:36.565066  7609 raft_consensus.cc:2781] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:36.565202  7609 raft_consensus.cc:482] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:36.565323  7609 raft_consensus.cc:3037] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:36.565181  7254 raft_consensus.cc:504] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:36.566372  7254 leader_election.cc:288] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399 [CANDIDATE]: Term 1 election: Requested vote from peers 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227), a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197)
I20240502 01:55:36.566387  6068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ed91df7ff04846991c95b30a86ea7e" candidate_uuid: "07cdb213c77c4ffda1f999b66b725399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218bc8c1f9ce46059898fc738179adf9"
I20240502 01:55:36.566608  6068 raft_consensus.cc:3037] T 73ed91df7ff04846991c95b30a86ea7e P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:36.567523  6204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ed91df7ff04846991c95b30a86ea7e" candidate_uuid: "07cdb213c77c4ffda1f999b66b725399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c90ee440b44b708bccc2e2d5385cc0"
I20240502 01:55:36.567775  6204 raft_consensus.cc:3037] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:36.568109  7609 raft_consensus.cc:504] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 01:55:36.568809  7609 leader_election.cc:288] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 [CANDIDATE]: Term 1 election: Requested vote from peers a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197), 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227)
I20240502 01:55:36.569339  6203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4486b413e5c14ec6953e4895ee5797cf" candidate_uuid: "07cdb213c77c4ffda1f999b66b725399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c90ee440b44b708bccc2e2d5385cc0"
I20240502 01:55:36.569418  6068 raft_consensus.cc:2454] T 73ed91df7ff04846991c95b30a86ea7e P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07cdb213c77c4ffda1f999b66b725399 in term 1.
I20240502 01:55:36.569563  6203 raft_consensus.cc:3037] T 4486b413e5c14ec6953e4895ee5797cf P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:36.570025  6204 raft_consensus.cc:2454] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07cdb213c77c4ffda1f999b66b725399 in term 1.
I20240502 01:55:36.570222  6402 leader_election.cc:302] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07cdb213c77c4ffda1f999b66b725399, 218bc8c1f9ce46059898fc738179adf9; no voters: 
I20240502 01:55:36.570555  7609 raft_consensus.cc:2781] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:36.570693  7609 raft_consensus.cc:686] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Becoming Leader. State: Replica: 07cdb213c77c4ffda1f999b66b725399, State: Running, Role: LEADER
I20240502 01:55:36.570973  6068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4486b413e5c14ec6953e4895ee5797cf" candidate_uuid: "07cdb213c77c4ffda1f999b66b725399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218bc8c1f9ce46059898fc738179adf9"
I20240502 01:55:36.570995  7609 consensus_queue.cc:237] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:36.571213  6068 raft_consensus.cc:3037] T 4486b413e5c14ec6953e4895ee5797cf P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:36.572315  6203 raft_consensus.cc:2454] T 4486b413e5c14ec6953e4895ee5797cf P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07cdb213c77c4ffda1f999b66b725399 in term 1.
I20240502 01:55:36.572856  6403 leader_election.cc:302] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07cdb213c77c4ffda1f999b66b725399, a2c90ee440b44b708bccc2e2d5385cc0; no voters: 
I20240502 01:55:36.573573  7609 raft_consensus.cc:2781] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:36.573745  7609 raft_consensus.cc:686] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Becoming Leader. State: Replica: 07cdb213c77c4ffda1f999b66b725399, State: Running, Role: LEADER
I20240502 01:55:36.573956  6068 raft_consensus.cc:2454] T 4486b413e5c14ec6953e4895ee5797cf P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07cdb213c77c4ffda1f999b66b725399 in term 1.
I20240502 01:55:36.574044  7609 consensus_queue.cc:237] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 01:55:36.574959  5922 catalog_manager.cc:5484] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132). New cstate: current_term: 1 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } }
I20240502 01:55:36.579953  5922 catalog_manager.cc:5484] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132). New cstate: current_term: 1 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } }
I20240502 01:55:36.950917  7534 consensus_queue.cc:1035] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:36.958714  7534 consensus_queue.cc:1035] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:55:36.985180  7613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:36.985535  7613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 01:55:36.989087  7254 consensus_queue.cc:1035] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:55:36.996819  7613 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:55:37.011317  7254 consensus_queue.cc:1035] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_6'
I20240502 01:55:37.111305  7624 consensus_queue.cc:1035] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:37.117808  7609 consensus_queue.cc:1035] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:55:37.244467  6402 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.129:36227, user_credentials={real_user=slave}} blocked reactor thread for 58723.3us

INSERT report
    rows total: 2000
    time total: 202.819 ms
  time per row: 0.101409 ms
I20240502 01:55:37.383635  5922 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58156:
name: "rebalance_test_table_7"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:55:37.393375  6315 tablet_service.cc:1467] Processing CreateTablet for tablet 4f4088f3523f4cd0afdf0de0f447c7af (DEFAULT_TABLE table=rebalance_test_table_7 [id=c2701645b8bc497aaf63e970157bdd80]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:37.393518  6047 tablet_service.cc:1467] Processing CreateTablet for tablet 4f4088f3523f4cd0afdf0de0f447c7af (DEFAULT_TABLE table=rebalance_test_table_7 [id=c2701645b8bc497aaf63e970157bdd80]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:37.393937  6315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f4088f3523f4cd0afdf0de0f447c7af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:37.393994  6047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f4088f3523f4cd0afdf0de0f447c7af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:37.393982  6182 tablet_service.cc:1467] Processing CreateTablet for tablet 4f4088f3523f4cd0afdf0de0f447c7af (DEFAULT_TABLE table=rebalance_test_table_7 [id=c2701645b8bc497aaf63e970157bdd80]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:37.394409  6182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f4088f3523f4cd0afdf0de0f447c7af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:37.396656  6446 tablet_service.cc:1467] Processing CreateTablet for tablet 0039dd405cfc4dcea5e7353169e3cd52 (DEFAULT_TABLE table=rebalance_test_table_7 [id=c2701645b8bc497aaf63e970157bdd80]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:37.397145  6446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0039dd405cfc4dcea5e7353169e3cd52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:37.400344  6447 tablet_service.cc:1467] Processing CreateTablet for tablet 265a9645e1f141cb88c786b9aef8f90e (DEFAULT_TABLE table=rebalance_test_table_7 [id=c2701645b8bc497aaf63e970157bdd80]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:37.400851  6447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 265a9645e1f141cb88c786b9aef8f90e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:37.401248  6048 tablet_service.cc:1467] Processing CreateTablet for tablet 265a9645e1f141cb88c786b9aef8f90e (DEFAULT_TABLE table=rebalance_test_table_7 [id=c2701645b8bc497aaf63e970157bdd80]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:37.401746  6048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 265a9645e1f141cb88c786b9aef8f90e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:37.405161  6180 tablet_service.cc:1467] Processing CreateTablet for tablet 0039dd405cfc4dcea5e7353169e3cd52 (DEFAULT_TABLE table=rebalance_test_table_7 [id=c2701645b8bc497aaf63e970157bdd80]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:37.405658  6180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0039dd405cfc4dcea5e7353169e3cd52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:37.405668  6316 tablet_service.cc:1467] Processing CreateTablet for tablet 265a9645e1f141cb88c786b9aef8f90e (DEFAULT_TABLE table=rebalance_test_table_7 [id=c2701645b8bc497aaf63e970157bdd80]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:37.406105  6316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 265a9645e1f141cb88c786b9aef8f90e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:37.410692  6314 tablet_service.cc:1467] Processing CreateTablet for tablet 0039dd405cfc4dcea5e7353169e3cd52 (DEFAULT_TABLE table=rebalance_test_table_7 [id=c2701645b8bc497aaf63e970157bdd80]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:37.411164  6314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0039dd405cfc4dcea5e7353169e3cd52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:37.412797  7680 tablet_bootstrap.cc:492] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9: Bootstrap starting.
I20240502 01:55:37.414003  7679 tablet_bootstrap.cc:492] T 4f4088f3523f4cd0afdf0de0f447c7af P a2c90ee440b44b708bccc2e2d5385cc0: Bootstrap starting.
I20240502 01:55:37.416018  7680 tablet_bootstrap.cc:654] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:37.416045  7679 tablet_bootstrap.cc:654] T 4f4088f3523f4cd0afdf0de0f447c7af P a2c90ee440b44b708bccc2e2d5385cc0: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:37.417997  7678 tablet_bootstrap.cc:492] T 4f4088f3523f4cd0afdf0de0f447c7af P 94ffd7aaf85b4e17881241b8396fa2f3: Bootstrap starting.
I20240502 01:55:37.420271  7678 tablet_bootstrap.cc:654] T 4f4088f3523f4cd0afdf0de0f447c7af P 94ffd7aaf85b4e17881241b8396fa2f3: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:37.422928  7678 tablet_bootstrap.cc:492] T 4f4088f3523f4cd0afdf0de0f447c7af P 94ffd7aaf85b4e17881241b8396fa2f3: No bootstrap required, opened a new log
I20240502 01:55:37.423744  7685 tablet_bootstrap.cc:492] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399: Bootstrap starting.
I20240502 01:55:37.423947  7680 tablet_bootstrap.cc:492] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9: No bootstrap required, opened a new log
I20240502 01:55:37.424129  7680 ts_tablet_manager.cc:1367] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9: Time spent bootstrapping tablet: real 0.011s	user 0.002s	sys 0.003s
I20240502 01:55:37.424248  7679 tablet_bootstrap.cc:492] T 4f4088f3523f4cd0afdf0de0f447c7af P a2c90ee440b44b708bccc2e2d5385cc0: No bootstrap required, opened a new log
I20240502 01:55:37.424413  7679 ts_tablet_manager.cc:1367] T 4f4088f3523f4cd0afdf0de0f447c7af P a2c90ee440b44b708bccc2e2d5385cc0: Time spent bootstrapping tablet: real 0.011s	user 0.001s	sys 0.009s
I20240502 01:55:37.425019  7680 raft_consensus.cc:348] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:37.425259  7680 raft_consensus.cc:374] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:37.425208  7679 raft_consensus.cc:348] T 4f4088f3523f4cd0afdf0de0f447c7af P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:37.425339  7680 raft_consensus.cc:729] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218bc8c1f9ce46059898fc738179adf9, State: Initialized, Role: FOLLOWER
I20240502 01:55:37.425401  7679 raft_consensus.cc:374] T 4f4088f3523f4cd0afdf0de0f447c7af P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:37.425412  7685 tablet_bootstrap.cc:654] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:37.425479  7679 raft_consensus.cc:729] T 4f4088f3523f4cd0afdf0de0f447c7af P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c90ee440b44b708bccc2e2d5385cc0, State: Initialized, Role: FOLLOWER
I20240502 01:55:37.425647  7680 consensus_queue.cc:260] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:37.425690  7679 consensus_queue.cc:260] T 4f4088f3523f4cd0afdf0de0f447c7af P a2c90ee440b44b708bccc2e2d5385cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:37.426277  7679 ts_tablet_manager.cc:1398] T 4f4088f3523f4cd0afdf0de0f447c7af P a2c90ee440b44b708bccc2e2d5385cc0: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:55:37.426568  7679 tablet_bootstrap.cc:492] T 0039dd405cfc4dcea5e7353169e3cd52 P a2c90ee440b44b708bccc2e2d5385cc0: Bootstrap starting.
I20240502 01:55:37.428272  7679 tablet_bootstrap.cc:654] T 0039dd405cfc4dcea5e7353169e3cd52 P a2c90ee440b44b708bccc2e2d5385cc0: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:37.428567  7680 ts_tablet_manager.cc:1398] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.002s
I20240502 01:55:37.428861  7680 tablet_bootstrap.cc:492] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9: Bootstrap starting.
I20240502 01:55:37.430716  7678 ts_tablet_manager.cc:1367] T 4f4088f3523f4cd0afdf0de0f447c7af P 94ffd7aaf85b4e17881241b8396fa2f3: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240502 01:55:37.431216  7679 tablet_bootstrap.cc:492] T 0039dd405cfc4dcea5e7353169e3cd52 P a2c90ee440b44b708bccc2e2d5385cc0: No bootstrap required, opened a new log
I20240502 01:55:37.431421  7680 tablet_bootstrap.cc:654] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:37.431416  7679 ts_tablet_manager.cc:1367] T 0039dd405cfc4dcea5e7353169e3cd52 P a2c90ee440b44b708bccc2e2d5385cc0: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240502 01:55:37.432353  7679 raft_consensus.cc:348] T 0039dd405cfc4dcea5e7353169e3cd52 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:37.432631  7679 raft_consensus.cc:374] T 0039dd405cfc4dcea5e7353169e3cd52 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:37.432739  7679 raft_consensus.cc:729] T 0039dd405cfc4dcea5e7353169e3cd52 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c90ee440b44b708bccc2e2d5385cc0, State: Initialized, Role: FOLLOWER
I20240502 01:55:37.432706  7678 raft_consensus.cc:348] T 4f4088f3523f4cd0afdf0de0f447c7af P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:37.433012  7679 consensus_queue.cc:260] T 0039dd405cfc4dcea5e7353169e3cd52 P a2c90ee440b44b708bccc2e2d5385cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:37.433353  7678 raft_consensus.cc:374] T 4f4088f3523f4cd0afdf0de0f447c7af P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:37.433502  7678 raft_consensus.cc:729] T 4f4088f3523f4cd0afdf0de0f447c7af P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94ffd7aaf85b4e17881241b8396fa2f3, State: Initialized, Role: FOLLOWER
I20240502 01:55:37.433737  7679 ts_tablet_manager.cc:1398] T 0039dd405cfc4dcea5e7353169e3cd52 P a2c90ee440b44b708bccc2e2d5385cc0: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 01:55:37.433765  7678 consensus_queue.cc:260] T 4f4088f3523f4cd0afdf0de0f447c7af P 94ffd7aaf85b4e17881241b8396fa2f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:37.434396  7678 ts_tablet_manager.cc:1398] T 4f4088f3523f4cd0afdf0de0f447c7af P 94ffd7aaf85b4e17881241b8396fa2f3: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:55:37.435838  7678 tablet_bootstrap.cc:492] T 265a9645e1f141cb88c786b9aef8f90e P 94ffd7aaf85b4e17881241b8396fa2f3: Bootstrap starting.
I20240502 01:55:37.437785  7678 tablet_bootstrap.cc:654] T 265a9645e1f141cb88c786b9aef8f90e P 94ffd7aaf85b4e17881241b8396fa2f3: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:37.438382  7680 tablet_bootstrap.cc:492] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9: No bootstrap required, opened a new log
I20240502 01:55:37.438589  7680 ts_tablet_manager.cc:1367] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9: Time spent bootstrapping tablet: real 0.010s	user 0.002s	sys 0.004s
I20240502 01:55:37.439512  7680 raft_consensus.cc:348] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 01:55:37.439776  7680 raft_consensus.cc:374] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:37.439853  7680 raft_consensus.cc:729] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218bc8c1f9ce46059898fc738179adf9, State: Initialized, Role: FOLLOWER
I20240502 01:55:37.440197  7680 consensus_queue.cc:260] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 01:55:37.440837  7680 ts_tablet_manager.cc:1398] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:55:37.444039  7678 tablet_bootstrap.cc:492] T 265a9645e1f141cb88c786b9aef8f90e P 94ffd7aaf85b4e17881241b8396fa2f3: No bootstrap required, opened a new log
I20240502 01:55:37.444077  7685 tablet_bootstrap.cc:492] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399: No bootstrap required, opened a new log
I20240502 01:55:37.444285  7685 ts_tablet_manager.cc:1367] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399: Time spent bootstrapping tablet: real 0.021s	user 0.005s	sys 0.015s
I20240502 01:55:37.444360  7678 ts_tablet_manager.cc:1367] T 265a9645e1f141cb88c786b9aef8f90e P 94ffd7aaf85b4e17881241b8396fa2f3: Time spent bootstrapping tablet: real 0.009s	user 0.001s	sys 0.006s
I20240502 01:55:37.445242  7678 raft_consensus.cc:348] T 265a9645e1f141cb88c786b9aef8f90e P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 01:55:37.445534  7678 raft_consensus.cc:374] T 265a9645e1f141cb88c786b9aef8f90e P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:37.445631  7678 raft_consensus.cc:729] T 265a9645e1f141cb88c786b9aef8f90e P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94ffd7aaf85b4e17881241b8396fa2f3, State: Initialized, Role: FOLLOWER
I20240502 01:55:37.445536  7685 raft_consensus.cc:348] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 01:55:37.445737  7685 raft_consensus.cc:374] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:37.445865  7685 raft_consensus.cc:729] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07cdb213c77c4ffda1f999b66b725399, State: Initialized, Role: FOLLOWER
I20240502 01:55:37.445922  7678 consensus_queue.cc:260] T 265a9645e1f141cb88c786b9aef8f90e P 94ffd7aaf85b4e17881241b8396fa2f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 01:55:37.446115  7685 consensus_queue.cc:260] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 01:55:37.446519  7678 ts_tablet_manager.cc:1398] T 265a9645e1f141cb88c786b9aef8f90e P 94ffd7aaf85b4e17881241b8396fa2f3: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 01:55:37.446895  7678 tablet_bootstrap.cc:492] T 0039dd405cfc4dcea5e7353169e3cd52 P 94ffd7aaf85b4e17881241b8396fa2f3: Bootstrap starting.
I20240502 01:55:37.447510  7685 ts_tablet_manager.cc:1398] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.001s
I20240502 01:55:37.447643  7673 raft_consensus.cc:482] T 0039dd405cfc4dcea5e7353169e3cd52 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:37.447942  7685 tablet_bootstrap.cc:492] T 0039dd405cfc4dcea5e7353169e3cd52 P 07cdb213c77c4ffda1f999b66b725399: Bootstrap starting.
I20240502 01:55:37.447831  7673 raft_consensus.cc:504] T 0039dd405cfc4dcea5e7353169e3cd52 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:37.448586  7673 leader_election.cc:288] T 0039dd405cfc4dcea5e7353169e3cd52 P a2c90ee440b44b708bccc2e2d5385cc0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245), 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867)
I20240502 01:55:37.448690  7678 tablet_bootstrap.cc:654] T 0039dd405cfc4dcea5e7353169e3cd52 P 94ffd7aaf85b4e17881241b8396fa2f3: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:37.449170  6334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0039dd405cfc4dcea5e7353169e3cd52" candidate_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" is_pre_election: true
I20240502 01:55:37.449558  7685 tablet_bootstrap.cc:654] T 0039dd405cfc4dcea5e7353169e3cd52 P 07cdb213c77c4ffda1f999b66b725399: Neither blocks nor log segments found. Creating new log.
W20240502 01:55:37.450131  6136 leader_election.cc:341] T 0039dd405cfc4dcea5e7353169e3cd52 P a2c90ee440b44b708bccc2e2d5385cc0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:55:37.450737  6467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0039dd405cfc4dcea5e7353169e3cd52" candidate_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07cdb213c77c4ffda1f999b66b725399" is_pre_election: true
W20240502 01:55:37.451472  6134 leader_election.cc:341] T 0039dd405cfc4dcea5e7353169e3cd52 P a2c90ee440b44b708bccc2e2d5385cc0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:55:37.451628  6134 leader_election.cc:302] T 0039dd405cfc4dcea5e7353169e3cd52 P a2c90ee440b44b708bccc2e2d5385cc0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2c90ee440b44b708bccc2e2d5385cc0; no voters: 07cdb213c77c4ffda1f999b66b725399, 94ffd7aaf85b4e17881241b8396fa2f3
I20240502 01:55:37.451809  7678 tablet_bootstrap.cc:492] T 0039dd405cfc4dcea5e7353169e3cd52 P 94ffd7aaf85b4e17881241b8396fa2f3: No bootstrap required, opened a new log
I20240502 01:55:37.452015  7678 ts_tablet_manager.cc:1367] T 0039dd405cfc4dcea5e7353169e3cd52 P 94ffd7aaf85b4e17881241b8396fa2f3: Time spent bootstrapping tablet: real 0.005s	user 0.005s	sys 0.000s
I20240502 01:55:37.452311  7685 tablet_bootstrap.cc:492] T 0039dd405cfc4dcea5e7353169e3cd52 P 07cdb213c77c4ffda1f999b66b725399: No bootstrap required, opened a new log
I20240502 01:55:37.452293  7673 raft_consensus.cc:2726] T 0039dd405cfc4dcea5e7353169e3cd52 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:55:37.452522  7685 ts_tablet_manager.cc:1367] T 0039dd405cfc4dcea5e7353169e3cd52 P 07cdb213c77c4ffda1f999b66b725399: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240502 01:55:37.452838  7678 raft_consensus.cc:348] T 0039dd405cfc4dcea5e7353169e3cd52 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:37.453105  7678 raft_consensus.cc:374] T 0039dd405cfc4dcea5e7353169e3cd52 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:37.453226  7678 raft_consensus.cc:729] T 0039dd405cfc4dcea5e7353169e3cd52 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94ffd7aaf85b4e17881241b8396fa2f3, State: Initialized, Role: FOLLOWER
I20240502 01:55:37.453285  7685 raft_consensus.cc:348] T 0039dd405cfc4dcea5e7353169e3cd52 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:37.453598  7685 raft_consensus.cc:374] T 0039dd405cfc4dcea5e7353169e3cd52 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:37.453526  7678 consensus_queue.cc:260] T 0039dd405cfc4dcea5e7353169e3cd52 P 94ffd7aaf85b4e17881241b8396fa2f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:37.453713  7685 raft_consensus.cc:729] T 0039dd405cfc4dcea5e7353169e3cd52 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07cdb213c77c4ffda1f999b66b725399, State: Initialized, Role: FOLLOWER
I20240502 01:55:37.454170  7678 ts_tablet_manager.cc:1398] T 0039dd405cfc4dcea5e7353169e3cd52 P 94ffd7aaf85b4e17881241b8396fa2f3: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:55:37.454159  7685 consensus_queue.cc:260] T 0039dd405cfc4dcea5e7353169e3cd52 P 07cdb213c77c4ffda1f999b66b725399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:37.454739  7685 ts_tablet_manager.cc:1398] T 0039dd405cfc4dcea5e7353169e3cd52 P 07cdb213c77c4ffda1f999b66b725399: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 01:55:37.475292  7615 raft_consensus.cc:482] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:37.475539  7615 raft_consensus.cc:504] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:37.476455  7615 leader_election.cc:288] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245), a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197)
I20240502 01:55:37.477018  6334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f4088f3523f4cd0afdf0de0f447c7af" candidate_uuid: "218bc8c1f9ce46059898fc738179adf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" is_pre_election: true
I20240502 01:55:37.477291  6334 raft_consensus.cc:2454] T 4f4088f3523f4cd0afdf0de0f447c7af P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 218bc8c1f9ce46059898fc738179adf9 in term 0.
I20240502 01:55:37.477319  6204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f4088f3523f4cd0afdf0de0f447c7af" candidate_uuid: "218bc8c1f9ce46059898fc738179adf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" is_pre_election: true
I20240502 01:55:37.477552  6204 raft_consensus.cc:2454] T 4f4088f3523f4cd0afdf0de0f447c7af P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 218bc8c1f9ce46059898fc738179adf9 in term 0.
I20240502 01:55:37.477846  6000 leader_election.cc:302] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9, 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 
I20240502 01:55:37.478241  7615 raft_consensus.cc:2781] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:37.478381  7615 raft_consensus.cc:482] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:37.478474  7615 raft_consensus.cc:3037] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:37.480728  7615 raft_consensus.cc:504] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:37.482123  6334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f4088f3523f4cd0afdf0de0f447c7af" candidate_uuid: "218bc8c1f9ce46059898fc738179adf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ffd7aaf85b4e17881241b8396fa2f3"
I20240502 01:55:37.482378  6334 raft_consensus.cc:3037] T 4f4088f3523f4cd0afdf0de0f447c7af P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:37.482740  6204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f4088f3523f4cd0afdf0de0f447c7af" candidate_uuid: "218bc8c1f9ce46059898fc738179adf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c90ee440b44b708bccc2e2d5385cc0"
I20240502 01:55:37.482961  6204 raft_consensus.cc:3037] T 4f4088f3523f4cd0afdf0de0f447c7af P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:37.485379  6334 raft_consensus.cc:2454] T 4f4088f3523f4cd0afdf0de0f447c7af P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 218bc8c1f9ce46059898fc738179adf9 in term 1.
I20240502 01:55:37.486191  6204 raft_consensus.cc:2454] T 4f4088f3523f4cd0afdf0de0f447c7af P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 218bc8c1f9ce46059898fc738179adf9 in term 1.
I20240502 01:55:37.486760  6001 leader_election.cc:302] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9, a2c90ee440b44b708bccc2e2d5385cc0; no voters: 
I20240502 01:55:37.487111  7658 raft_consensus.cc:2781] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:37.487277  7658 raft_consensus.cc:686] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 [term 1 LEADER]: Becoming Leader. State: Replica: 218bc8c1f9ce46059898fc738179adf9, State: Running, Role: LEADER
I20240502 01:55:37.487200  7615 leader_election.cc:288] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 1 election: Requested vote from peers 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245), a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197)
I20240502 01:55:37.487617  7658 consensus_queue.cc:237] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:37.491436  5921 catalog_manager.cc:5484] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129). New cstate: current_term: 1 leader_uuid: "218bc8c1f9ce46059898fc738179adf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } }
I20240502 01:55:37.709120  7624 raft_consensus.cc:482] T 0039dd405cfc4dcea5e7353169e3cd52 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:37.709385  7624 raft_consensus.cc:504] T 0039dd405cfc4dcea5e7353169e3cd52 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:37.710074  7624 leader_election.cc:288] T 0039dd405cfc4dcea5e7353169e3cd52 P 07cdb213c77c4ffda1f999b66b725399 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245), a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197)
I20240502 01:55:37.710794  6334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0039dd405cfc4dcea5e7353169e3cd52" candidate_uuid: "07cdb213c77c4ffda1f999b66b725399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" is_pre_election: true
I20240502 01:55:37.710809  6204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0039dd405cfc4dcea5e7353169e3cd52" candidate_uuid: "07cdb213c77c4ffda1f999b66b725399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" is_pre_election: true
I20240502 01:55:37.711087  6204 raft_consensus.cc:2454] T 0039dd405cfc4dcea5e7353169e3cd52 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07cdb213c77c4ffda1f999b66b725399 in term 0.
I20240502 01:55:37.711115  6334 raft_consensus.cc:2454] T 0039dd405cfc4dcea5e7353169e3cd52 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07cdb213c77c4ffda1f999b66b725399 in term 0.
I20240502 01:55:37.711541  6403 leader_election.cc:302] T 0039dd405cfc4dcea5e7353169e3cd52 P 07cdb213c77c4ffda1f999b66b725399 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07cdb213c77c4ffda1f999b66b725399, a2c90ee440b44b708bccc2e2d5385cc0; no voters: 
I20240502 01:55:37.711915  7624 raft_consensus.cc:2781] T 0039dd405cfc4dcea5e7353169e3cd52 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:37.712035  7624 raft_consensus.cc:482] T 0039dd405cfc4dcea5e7353169e3cd52 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:37.712150  7624 raft_consensus.cc:3037] T 0039dd405cfc4dcea5e7353169e3cd52 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:37.714089  7624 raft_consensus.cc:504] T 0039dd405cfc4dcea5e7353169e3cd52 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:37.714674  7624 leader_election.cc:288] T 0039dd405cfc4dcea5e7353169e3cd52 P 07cdb213c77c4ffda1f999b66b725399 [CANDIDATE]: Term 1 election: Requested vote from peers 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245), a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197)
I20240502 01:55:37.715241  6334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0039dd405cfc4dcea5e7353169e3cd52" candidate_uuid: "07cdb213c77c4ffda1f999b66b725399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ffd7aaf85b4e17881241b8396fa2f3"
I20240502 01:55:37.715516  6334 raft_consensus.cc:3037] T 0039dd405cfc4dcea5e7353169e3cd52 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:37.715497  6204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0039dd405cfc4dcea5e7353169e3cd52" candidate_uuid: "07cdb213c77c4ffda1f999b66b725399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c90ee440b44b708bccc2e2d5385cc0"
I20240502 01:55:37.715744  6204 raft_consensus.cc:3037] T 0039dd405cfc4dcea5e7353169e3cd52 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:37.718456  6334 raft_consensus.cc:2454] T 0039dd405cfc4dcea5e7353169e3cd52 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07cdb213c77c4ffda1f999b66b725399 in term 1.
I20240502 01:55:37.718483  6204 raft_consensus.cc:2454] T 0039dd405cfc4dcea5e7353169e3cd52 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07cdb213c77c4ffda1f999b66b725399 in term 1.
I20240502 01:55:37.719000  6402 leader_election.cc:302] T 0039dd405cfc4dcea5e7353169e3cd52 P 07cdb213c77c4ffda1f999b66b725399 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07cdb213c77c4ffda1f999b66b725399, 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 
I20240502 01:55:37.719342  7624 raft_consensus.cc:2781] T 0039dd405cfc4dcea5e7353169e3cd52 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:37.719538  7624 raft_consensus.cc:686] T 0039dd405cfc4dcea5e7353169e3cd52 P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Becoming Leader. State: Replica: 07cdb213c77c4ffda1f999b66b725399, State: Running, Role: LEADER
I20240502 01:55:37.719812  7624 consensus_queue.cc:237] T 0039dd405cfc4dcea5e7353169e3cd52 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:37.722846  5921 catalog_manager.cc:5484] T 0039dd405cfc4dcea5e7353169e3cd52 P 07cdb213c77c4ffda1f999b66b725399 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132). New cstate: current_term: 1 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } }
I20240502 01:55:37.784353  7624 raft_consensus.cc:482] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:37.784559  7624 raft_consensus.cc:504] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 01:55:37.785189  7624 leader_election.cc:288] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245), 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227)
I20240502 01:55:37.785765  6334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "265a9645e1f141cb88c786b9aef8f90e" candidate_uuid: "07cdb213c77c4ffda1f999b66b725399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" is_pre_election: true
I20240502 01:55:37.785820  6068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "265a9645e1f141cb88c786b9aef8f90e" candidate_uuid: "07cdb213c77c4ffda1f999b66b725399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218bc8c1f9ce46059898fc738179adf9" is_pre_election: true
I20240502 01:55:37.785985  6334 raft_consensus.cc:2454] T 265a9645e1f141cb88c786b9aef8f90e P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07cdb213c77c4ffda1f999b66b725399 in term 0.
I20240502 01:55:37.786091  6068 raft_consensus.cc:2454] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07cdb213c77c4ffda1f999b66b725399 in term 0.
I20240502 01:55:37.786437  6402 leader_election.cc:302] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07cdb213c77c4ffda1f999b66b725399, 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 
I20240502 01:55:37.786741  7624 raft_consensus.cc:2781] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:37.786859  7624 raft_consensus.cc:482] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:37.786945  7624 raft_consensus.cc:3037] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:37.789232  7624 raft_consensus.cc:504] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 01:55:37.789796  7624 leader_election.cc:288] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399 [CANDIDATE]: Term 1 election: Requested vote from peers 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245), 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227)
I20240502 01:55:37.790354  6334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "265a9645e1f141cb88c786b9aef8f90e" candidate_uuid: "07cdb213c77c4ffda1f999b66b725399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ffd7aaf85b4e17881241b8396fa2f3"
I20240502 01:55:37.790387  6068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "265a9645e1f141cb88c786b9aef8f90e" candidate_uuid: "07cdb213c77c4ffda1f999b66b725399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218bc8c1f9ce46059898fc738179adf9"
I20240502 01:55:37.790570  6334 raft_consensus.cc:3037] T 265a9645e1f141cb88c786b9aef8f90e P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:37.790585  6068 raft_consensus.cc:3037] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:37.792483  6068 raft_consensus.cc:2454] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07cdb213c77c4ffda1f999b66b725399 in term 1.
I20240502 01:55:37.792573  6334 raft_consensus.cc:2454] T 265a9645e1f141cb88c786b9aef8f90e P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07cdb213c77c4ffda1f999b66b725399 in term 1.
I20240502 01:55:37.792912  6402 leader_election.cc:302] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07cdb213c77c4ffda1f999b66b725399, 218bc8c1f9ce46059898fc738179adf9; no voters: 
I20240502 01:55:37.793252  7624 raft_consensus.cc:2781] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:37.793396  7624 raft_consensus.cc:686] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Becoming Leader. State: Replica: 07cdb213c77c4ffda1f999b66b725399, State: Running, Role: LEADER
I20240502 01:55:37.793663  7624 consensus_queue.cc:237] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 01:55:37.796659  5921 catalog_manager.cc:5484] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132). New cstate: current_term: 1 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } }
I20240502 01:55:37.982553  7658 consensus_queue.cc:1035] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:37.988112  7534 consensus_queue.cc:1035] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:38.100193  7624 consensus_queue.cc:1035] T 0039dd405cfc4dcea5e7353169e3cd52 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:38.106443  7624 consensus_queue.cc:1035] T 0039dd405cfc4dcea5e7353169e3cd52 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:38.224530  7624 consensus_queue.cc:1035] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:38.229671  7624 consensus_queue.cc:1035] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:55:38.261623  7697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:38.261981  7697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:38.270437  7697 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'

INSERT report
    rows total: 2000
    time total: 194.534 ms
  time per row: 0.0972672 ms
I20240502 01:55:38.622752  5921 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58156:
name: "rebalance_test_table_8"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:55:38.632015  6182 tablet_service.cc:1467] Processing CreateTablet for tablet 3a7698f6bd4349ae832a22c853d76fea (DEFAULT_TABLE table=rebalance_test_table_8 [id=fcde569cfdf747c2aaecac7d265b4b1f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:38.632532  6182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a7698f6bd4349ae832a22c853d76fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:38.632486  6315 tablet_service.cc:1467] Processing CreateTablet for tablet 3a7698f6bd4349ae832a22c853d76fea (DEFAULT_TABLE table=rebalance_test_table_8 [id=fcde569cfdf747c2aaecac7d265b4b1f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:38.632907  6315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a7698f6bd4349ae832a22c853d76fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:38.633505  6047 tablet_service.cc:1467] Processing CreateTablet for tablet 3a7698f6bd4349ae832a22c853d76fea (DEFAULT_TABLE table=rebalance_test_table_8 [id=fcde569cfdf747c2aaecac7d265b4b1f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:38.633963  6047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a7698f6bd4349ae832a22c853d76fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:38.633968  6048 tablet_service.cc:1467] Processing CreateTablet for tablet 99e74a49b42745888058db67049dc682 (DEFAULT_TABLE table=rebalance_test_table_8 [id=fcde569cfdf747c2aaecac7d265b4b1f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:38.634589  6048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99e74a49b42745888058db67049dc682. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:38.636320  6046 tablet_service.cc:1467] Processing CreateTablet for tablet d8a6db6ae11e4cb8881aee95bdb0e59d (DEFAULT_TABLE table=rebalance_test_table_8 [id=fcde569cfdf747c2aaecac7d265b4b1f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:38.636698  6046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8a6db6ae11e4cb8881aee95bdb0e59d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:38.639758  6447 tablet_service.cc:1467] Processing CreateTablet for tablet 99e74a49b42745888058db67049dc682 (DEFAULT_TABLE table=rebalance_test_table_8 [id=fcde569cfdf747c2aaecac7d265b4b1f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:38.640200  6447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99e74a49b42745888058db67049dc682. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:38.640308  6314 tablet_service.cc:1467] Processing CreateTablet for tablet d8a6db6ae11e4cb8881aee95bdb0e59d (DEFAULT_TABLE table=rebalance_test_table_8 [id=fcde569cfdf747c2aaecac7d265b4b1f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:38.640734  6314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8a6db6ae11e4cb8881aee95bdb0e59d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:38.644454  7752 tablet_bootstrap.cc:492] T 3a7698f6bd4349ae832a22c853d76fea P a2c90ee440b44b708bccc2e2d5385cc0: Bootstrap starting.
I20240502 01:55:38.646286  7752 tablet_bootstrap.cc:654] T 3a7698f6bd4349ae832a22c853d76fea P a2c90ee440b44b708bccc2e2d5385cc0: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:38.639793  6446 tablet_service.cc:1467] Processing CreateTablet for tablet d8a6db6ae11e4cb8881aee95bdb0e59d (DEFAULT_TABLE table=rebalance_test_table_8 [id=fcde569cfdf747c2aaecac7d265b4b1f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:38.646837  6446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8a6db6ae11e4cb8881aee95bdb0e59d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:38.649262  6182 tablet_service.cc:1467] Processing CreateTablet for tablet 99e74a49b42745888058db67049dc682 (DEFAULT_TABLE table=rebalance_test_table_8 [id=fcde569cfdf747c2aaecac7d265b4b1f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:38.649703  6182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99e74a49b42745888058db67049dc682. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:38.650341  7753 tablet_bootstrap.cc:492] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9: Bootstrap starting.
I20240502 01:55:38.653120  7753 tablet_bootstrap.cc:654] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:38.663254  7753 tablet_bootstrap.cc:492] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9: No bootstrap required, opened a new log
I20240502 01:55:38.663571  7753 ts_tablet_manager.cc:1367] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9: Time spent bootstrapping tablet: real 0.013s	user 0.008s	sys 0.004s
I20240502 01:55:38.664633  7753 raft_consensus.cc:348] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:38.665019  7757 tablet_bootstrap.cc:492] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3: Bootstrap starting.
I20240502 01:55:38.665030  7753 raft_consensus.cc:374] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:38.665128  7753 raft_consensus.cc:729] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218bc8c1f9ce46059898fc738179adf9, State: Initialized, Role: FOLLOWER
I20240502 01:55:38.665457  7753 consensus_queue.cc:260] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:38.667057  7753 ts_tablet_manager.cc:1398] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 01:55:38.667435  7753 tablet_bootstrap.cc:492] T 3a7698f6bd4349ae832a22c853d76fea P 218bc8c1f9ce46059898fc738179adf9: Bootstrap starting.
I20240502 01:55:38.669224  7753 tablet_bootstrap.cc:654] T 3a7698f6bd4349ae832a22c853d76fea P 218bc8c1f9ce46059898fc738179adf9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:38.670403  7757 tablet_bootstrap.cc:654] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:38.672259  7755 tablet_bootstrap.cc:492] T 99e74a49b42745888058db67049dc682 P 07cdb213c77c4ffda1f999b66b725399: Bootstrap starting.
I20240502 01:55:38.672657  7752 tablet_bootstrap.cc:492] T 3a7698f6bd4349ae832a22c853d76fea P a2c90ee440b44b708bccc2e2d5385cc0: No bootstrap required, opened a new log
I20240502 01:55:38.672847  7752 ts_tablet_manager.cc:1367] T 3a7698f6bd4349ae832a22c853d76fea P a2c90ee440b44b708bccc2e2d5385cc0: Time spent bootstrapping tablet: real 0.029s	user 0.008s	sys 0.012s
I20240502 01:55:38.673528  7752 raft_consensus.cc:348] T 3a7698f6bd4349ae832a22c853d76fea P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:38.673712  7752 raft_consensus.cc:374] T 3a7698f6bd4349ae832a22c853d76fea P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:38.673839  7752 raft_consensus.cc:729] T 3a7698f6bd4349ae832a22c853d76fea P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c90ee440b44b708bccc2e2d5385cc0, State: Initialized, Role: FOLLOWER
I20240502 01:55:38.674078  7752 consensus_queue.cc:260] T 3a7698f6bd4349ae832a22c853d76fea P a2c90ee440b44b708bccc2e2d5385cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:38.674566  7752 ts_tablet_manager.cc:1398] T 3a7698f6bd4349ae832a22c853d76fea P a2c90ee440b44b708bccc2e2d5385cc0: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240502 01:55:38.674829  7752 tablet_bootstrap.cc:492] T 99e74a49b42745888058db67049dc682 P a2c90ee440b44b708bccc2e2d5385cc0: Bootstrap starting.
I20240502 01:55:38.675276  7755 tablet_bootstrap.cc:654] T 99e74a49b42745888058db67049dc682 P 07cdb213c77c4ffda1f999b66b725399: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:38.676375  7752 tablet_bootstrap.cc:654] T 99e74a49b42745888058db67049dc682 P a2c90ee440b44b708bccc2e2d5385cc0: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:38.677307  7753 tablet_bootstrap.cc:492] T 3a7698f6bd4349ae832a22c853d76fea P 218bc8c1f9ce46059898fc738179adf9: No bootstrap required, opened a new log
I20240502 01:55:38.677592  7753 ts_tablet_manager.cc:1367] T 3a7698f6bd4349ae832a22c853d76fea P 218bc8c1f9ce46059898fc738179adf9: Time spent bootstrapping tablet: real 0.010s	user 0.003s	sys 0.000s
I20240502 01:55:38.679726  7753 raft_consensus.cc:348] T 3a7698f6bd4349ae832a22c853d76fea P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:38.680109  7753 raft_consensus.cc:374] T 3a7698f6bd4349ae832a22c853d76fea P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:38.680227  7753 raft_consensus.cc:729] T 3a7698f6bd4349ae832a22c853d76fea P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218bc8c1f9ce46059898fc738179adf9, State: Initialized, Role: FOLLOWER
I20240502 01:55:38.680621  7753 consensus_queue.cc:260] T 3a7698f6bd4349ae832a22c853d76fea P 218bc8c1f9ce46059898fc738179adf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:38.682237  7753 ts_tablet_manager.cc:1398] T 3a7698f6bd4349ae832a22c853d76fea P 218bc8c1f9ce46059898fc738179adf9: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 01:55:38.682627  7753 tablet_bootstrap.cc:492] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9: Bootstrap starting.
I20240502 01:55:38.684651  7752 tablet_bootstrap.cc:492] T 99e74a49b42745888058db67049dc682 P a2c90ee440b44b708bccc2e2d5385cc0: No bootstrap required, opened a new log
I20240502 01:55:38.684846  7752 ts_tablet_manager.cc:1367] T 99e74a49b42745888058db67049dc682 P a2c90ee440b44b708bccc2e2d5385cc0: Time spent bootstrapping tablet: real 0.010s	user 0.009s	sys 0.000s
I20240502 01:55:38.685572  7753 tablet_bootstrap.cc:654] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:38.685552  7752 raft_consensus.cc:348] T 99e74a49b42745888058db67049dc682 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:38.685884  7752 raft_consensus.cc:374] T 99e74a49b42745888058db67049dc682 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:38.686018  7752 raft_consensus.cc:729] T 99e74a49b42745888058db67049dc682 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c90ee440b44b708bccc2e2d5385cc0, State: Initialized, Role: FOLLOWER
I20240502 01:55:38.686370  7752 consensus_queue.cc:260] T 99e74a49b42745888058db67049dc682 P a2c90ee440b44b708bccc2e2d5385cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:38.687177  7752 ts_tablet_manager.cc:1398] T 99e74a49b42745888058db67049dc682 P a2c90ee440b44b708bccc2e2d5385cc0: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
I20240502 01:55:38.688709  7755 tablet_bootstrap.cc:492] T 99e74a49b42745888058db67049dc682 P 07cdb213c77c4ffda1f999b66b725399: No bootstrap required, opened a new log
I20240502 01:55:38.688963  7755 ts_tablet_manager.cc:1367] T 99e74a49b42745888058db67049dc682 P 07cdb213c77c4ffda1f999b66b725399: Time spent bootstrapping tablet: real 0.017s	user 0.002s	sys 0.002s
I20240502 01:55:38.690948  7755 raft_consensus.cc:348] T 99e74a49b42745888058db67049dc682 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:38.691215  7755 raft_consensus.cc:374] T 99e74a49b42745888058db67049dc682 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:38.691337  7755 raft_consensus.cc:729] T 99e74a49b42745888058db67049dc682 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07cdb213c77c4ffda1f999b66b725399, State: Initialized, Role: FOLLOWER
I20240502 01:55:38.691905  7755 consensus_queue.cc:260] T 99e74a49b42745888058db67049dc682 P 07cdb213c77c4ffda1f999b66b725399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:38.692672  7755 ts_tablet_manager.cc:1398] T 99e74a49b42745888058db67049dc682 P 07cdb213c77c4ffda1f999b66b725399: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 01:55:38.693035  7755 tablet_bootstrap.cc:492] T d8a6db6ae11e4cb8881aee95bdb0e59d P 07cdb213c77c4ffda1f999b66b725399: Bootstrap starting.
I20240502 01:55:38.693439  7753 tablet_bootstrap.cc:492] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9: No bootstrap required, opened a new log
I20240502 01:55:38.693619  7753 ts_tablet_manager.cc:1367] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9: Time spent bootstrapping tablet: real 0.011s	user 0.003s	sys 0.000s
I20240502 01:55:38.693769  7757 tablet_bootstrap.cc:492] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3: No bootstrap required, opened a new log
I20240502 01:55:38.693989  7757 ts_tablet_manager.cc:1367] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3: Time spent bootstrapping tablet: real 0.029s	user 0.004s	sys 0.014s
I20240502 01:55:38.694486  7753 raft_consensus.cc:348] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:38.694716  7753 raft_consensus.cc:374] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:38.694824  7755 tablet_bootstrap.cc:654] T d8a6db6ae11e4cb8881aee95bdb0e59d P 07cdb213c77c4ffda1f999b66b725399: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:38.694814  7753 raft_consensus.cc:729] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218bc8c1f9ce46059898fc738179adf9, State: Initialized, Role: FOLLOWER
I20240502 01:55:38.695053  7757 raft_consensus.cc:348] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:38.695099  7753 consensus_queue.cc:260] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:38.695353  7757 raft_consensus.cc:374] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:38.695495  7757 raft_consensus.cc:729] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94ffd7aaf85b4e17881241b8396fa2f3, State: Initialized, Role: FOLLOWER
I20240502 01:55:38.695820  7753 ts_tablet_manager.cc:1398] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 01:55:38.696250  7757 consensus_queue.cc:260] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:38.697664  7757 ts_tablet_manager.cc:1398] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.001s
I20240502 01:55:38.698601  7757 tablet_bootstrap.cc:492] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3: Bootstrap starting.
I20240502 01:55:38.700431  7757 tablet_bootstrap.cc:654] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:38.702049  7755 tablet_bootstrap.cc:492] T d8a6db6ae11e4cb8881aee95bdb0e59d P 07cdb213c77c4ffda1f999b66b725399: No bootstrap required, opened a new log
I20240502 01:55:38.702240  7755 ts_tablet_manager.cc:1367] T d8a6db6ae11e4cb8881aee95bdb0e59d P 07cdb213c77c4ffda1f999b66b725399: Time spent bootstrapping tablet: real 0.009s	user 0.002s	sys 0.001s
I20240502 01:55:38.702945  7755 raft_consensus.cc:348] T d8a6db6ae11e4cb8881aee95bdb0e59d P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:38.703145  7755 raft_consensus.cc:374] T d8a6db6ae11e4cb8881aee95bdb0e59d P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:38.703215  7755 raft_consensus.cc:729] T d8a6db6ae11e4cb8881aee95bdb0e59d P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07cdb213c77c4ffda1f999b66b725399, State: Initialized, Role: FOLLOWER
I20240502 01:55:38.703454  7755 consensus_queue.cc:260] T d8a6db6ae11e4cb8881aee95bdb0e59d P 07cdb213c77c4ffda1f999b66b725399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:38.703969  7755 ts_tablet_manager.cc:1398] T d8a6db6ae11e4cb8881aee95bdb0e59d P 07cdb213c77c4ffda1f999b66b725399: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 01:55:38.705262  7757 tablet_bootstrap.cc:492] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3: No bootstrap required, opened a new log
I20240502 01:55:38.705556  7757 ts_tablet_manager.cc:1367] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3: Time spent bootstrapping tablet: real 0.007s	user 0.003s	sys 0.000s
I20240502 01:55:38.706236  7757 raft_consensus.cc:348] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:38.706439  7757 raft_consensus.cc:374] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:38.706509  7757 raft_consensus.cc:729] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94ffd7aaf85b4e17881241b8396fa2f3, State: Initialized, Role: FOLLOWER
I20240502 01:55:38.706759  7757 consensus_queue.cc:260] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:38.707312  7757 ts_tablet_manager.cc:1398] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:55:38.737593  7534 raft_consensus.cc:482] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:38.737831  7534 raft_consensus.cc:504] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:38.738556  7534 leader_election.cc:288] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867), a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197)
I20240502 01:55:38.739071  6469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e74a49b42745888058db67049dc682" candidate_uuid: "218bc8c1f9ce46059898fc738179adf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07cdb213c77c4ffda1f999b66b725399" is_pre_election: true
I20240502 01:55:38.739132  6203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e74a49b42745888058db67049dc682" candidate_uuid: "218bc8c1f9ce46059898fc738179adf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" is_pre_election: true
I20240502 01:55:38.739343  6469 raft_consensus.cc:2454] T 99e74a49b42745888058db67049dc682 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 218bc8c1f9ce46059898fc738179adf9 in term 0.
I20240502 01:55:38.739377  6203 raft_consensus.cc:2454] T 99e74a49b42745888058db67049dc682 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 218bc8c1f9ce46059898fc738179adf9 in term 0.
I20240502 01:55:38.739835  6001 leader_election.cc:302] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9, a2c90ee440b44b708bccc2e2d5385cc0; no voters: 
I20240502 01:55:38.740294  7534 raft_consensus.cc:2781] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:38.740406  7534 raft_consensus.cc:482] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:38.740530  7534 raft_consensus.cc:3037] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:38.742358  7534 raft_consensus.cc:504] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:38.742905  7534 leader_election.cc:288] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 1 election: Requested vote from peers 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867), a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197)
I20240502 01:55:38.743347  6469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e74a49b42745888058db67049dc682" candidate_uuid: "218bc8c1f9ce46059898fc738179adf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07cdb213c77c4ffda1f999b66b725399"
I20240502 01:55:38.743350  6203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e74a49b42745888058db67049dc682" candidate_uuid: "218bc8c1f9ce46059898fc738179adf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c90ee440b44b708bccc2e2d5385cc0"
I20240502 01:55:38.743541  6203 raft_consensus.cc:3037] T 99e74a49b42745888058db67049dc682 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:38.743553  6469 raft_consensus.cc:3037] T 99e74a49b42745888058db67049dc682 P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:38.745651  6469 raft_consensus.cc:2454] T 99e74a49b42745888058db67049dc682 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 218bc8c1f9ce46059898fc738179adf9 in term 1.
I20240502 01:55:38.745679  6203 raft_consensus.cc:2454] T 99e74a49b42745888058db67049dc682 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 218bc8c1f9ce46059898fc738179adf9 in term 1.
I20240502 01:55:38.746089  6001 leader_election.cc:302] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9, a2c90ee440b44b708bccc2e2d5385cc0; no voters: 
I20240502 01:55:38.746410  7534 raft_consensus.cc:2781] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:38.746593  7534 raft_consensus.cc:686] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 [term 1 LEADER]: Becoming Leader. State: Replica: 218bc8c1f9ce46059898fc738179adf9, State: Running, Role: LEADER
I20240502 01:55:38.746879  7534 consensus_queue.cc:237] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 01:55:38.750164  5916 catalog_manager.cc:5484] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129). New cstate: current_term: 1 leader_uuid: "218bc8c1f9ce46059898fc738179adf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } }
I20240502 01:55:38.786836  7534 raft_consensus.cc:482] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:38.787060  7534 raft_consensus.cc:504] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:38.787789  7534 leader_election.cc:288] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245), 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867)
I20240502 01:55:38.788427  6336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a6db6ae11e4cb8881aee95bdb0e59d" candidate_uuid: "218bc8c1f9ce46059898fc738179adf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" is_pre_election: true
I20240502 01:55:38.788451  6469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a6db6ae11e4cb8881aee95bdb0e59d" candidate_uuid: "218bc8c1f9ce46059898fc738179adf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07cdb213c77c4ffda1f999b66b725399" is_pre_election: true
I20240502 01:55:38.788667  6336 raft_consensus.cc:2454] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 218bc8c1f9ce46059898fc738179adf9 in term 0.
I20240502 01:55:38.788683  6469 raft_consensus.cc:2454] T d8a6db6ae11e4cb8881aee95bdb0e59d P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 218bc8c1f9ce46059898fc738179adf9 in term 0.
I20240502 01:55:38.789101  5998 leader_election.cc:302] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07cdb213c77c4ffda1f999b66b725399, 218bc8c1f9ce46059898fc738179adf9; no voters: 
I20240502 01:55:38.789374  7534 raft_consensus.cc:2781] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:38.789495  7534 raft_consensus.cc:482] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:38.789569  7534 raft_consensus.cc:3037] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:38.791484  7534 raft_consensus.cc:504] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:38.792124  7534 leader_election.cc:288] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 1 election: Requested vote from peers 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245), 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867)
I20240502 01:55:38.792624  6336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a6db6ae11e4cb8881aee95bdb0e59d" candidate_uuid: "218bc8c1f9ce46059898fc738179adf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ffd7aaf85b4e17881241b8396fa2f3"
I20240502 01:55:38.792742  6469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a6db6ae11e4cb8881aee95bdb0e59d" candidate_uuid: "218bc8c1f9ce46059898fc738179adf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07cdb213c77c4ffda1f999b66b725399"
I20240502 01:55:38.792860  6336 raft_consensus.cc:3037] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:38.792959  6469 raft_consensus.cc:3037] T d8a6db6ae11e4cb8881aee95bdb0e59d P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:38.795769  6469 raft_consensus.cc:2454] T d8a6db6ae11e4cb8881aee95bdb0e59d P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 218bc8c1f9ce46059898fc738179adf9 in term 1.
I20240502 01:55:38.795769  6336 raft_consensus.cc:2454] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 218bc8c1f9ce46059898fc738179adf9 in term 1.
I20240502 01:55:38.796429  5998 leader_election.cc:302] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07cdb213c77c4ffda1f999b66b725399, 218bc8c1f9ce46059898fc738179adf9; no voters: 
I20240502 01:55:38.796821  7534 raft_consensus.cc:2781] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:38.796989  7534 raft_consensus.cc:686] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9 [term 1 LEADER]: Becoming Leader. State: Replica: 218bc8c1f9ce46059898fc738179adf9, State: Running, Role: LEADER
I20240502 01:55:38.797282  7534 consensus_queue.cc:237] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:38.800652  5916 catalog_manager.cc:5484] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129). New cstate: current_term: 1 leader_uuid: "218bc8c1f9ce46059898fc738179adf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } }
I20240502 01:55:38.967264  7695 raft_consensus.cc:482] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:38.967465  7695 raft_consensus.cc:504] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:38.968235  7695 leader_election.cc:288] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197), 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227)
I20240502 01:55:38.968801  6067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7698f6bd4349ae832a22c853d76fea" candidate_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218bc8c1f9ce46059898fc738179adf9" is_pre_election: true
I20240502 01:55:38.968791  6203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7698f6bd4349ae832a22c853d76fea" candidate_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" is_pre_election: true
I20240502 01:55:38.969033  6067 raft_consensus.cc:2454] T 3a7698f6bd4349ae832a22c853d76fea P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94ffd7aaf85b4e17881241b8396fa2f3 in term 0.
I20240502 01:55:38.969048  6203 raft_consensus.cc:2454] T 3a7698f6bd4349ae832a22c853d76fea P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94ffd7aaf85b4e17881241b8396fa2f3 in term 0.
I20240502 01:55:38.969496  6271 leader_election.cc:302] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94ffd7aaf85b4e17881241b8396fa2f3, a2c90ee440b44b708bccc2e2d5385cc0; no voters: 
I20240502 01:55:38.969801  7695 raft_consensus.cc:2781] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:38.969908  7695 raft_consensus.cc:482] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:38.969972  7695 raft_consensus.cc:3037] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:38.971906  7695 raft_consensus.cc:504] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:38.972537  7695 leader_election.cc:288] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 1 election: Requested vote from peers a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197), 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227)
I20240502 01:55:38.973141  6203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7698f6bd4349ae832a22c853d76fea" candidate_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c90ee440b44b708bccc2e2d5385cc0"
I20240502 01:55:38.973181  6067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a7698f6bd4349ae832a22c853d76fea" candidate_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218bc8c1f9ce46059898fc738179adf9"
I20240502 01:55:38.973381  6067 raft_consensus.cc:3037] T 3a7698f6bd4349ae832a22c853d76fea P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:38.973402  6203 raft_consensus.cc:3037] T 3a7698f6bd4349ae832a22c853d76fea P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:38.976264  6067 raft_consensus.cc:2454] T 3a7698f6bd4349ae832a22c853d76fea P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ffd7aaf85b4e17881241b8396fa2f3 in term 1.
I20240502 01:55:38.976249  6203 raft_consensus.cc:2454] T 3a7698f6bd4349ae832a22c853d76fea P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ffd7aaf85b4e17881241b8396fa2f3 in term 1.
I20240502 01:55:38.976819  6270 leader_election.cc:302] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9, 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 
I20240502 01:55:38.977172  7695 raft_consensus.cc:2781] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:38.977623  7695 raft_consensus.cc:686] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Becoming Leader. State: Replica: 94ffd7aaf85b4e17881241b8396fa2f3, State: Running, Role: LEADER
I20240502 01:55:38.977982  7695 consensus_queue.cc:237] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 01:55:38.981101  5916 catalog_manager.cc:5484] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131). New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } }
I20240502 01:55:39.175609  7534 consensus_queue.cc:1035] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:39.187815  7534 consensus_queue.cc:1035] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:39.200208  7615 consensus_queue.cc:1035] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:39.211488  7775 consensus_queue.cc:1035] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:55:39.403970  7771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:39.404325  7771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:39.412379  7771 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240502 01:55:39.486125  7769 consensus_queue.cc:1035] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:39.510602  7695 consensus_queue.cc:1035] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 204.325 ms
  time per row: 0.102162 ms
I20240502 01:55:39.803524  5921 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58156:
name: "rebalance_test_table_9"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 01:55:39.813959  6315 tablet_service.cc:1467] Processing CreateTablet for tablet 2cefdfa8851147999c1425c606f36b1f (DEFAULT_TABLE table=rebalance_test_table_9 [id=f406d4a60a5e40c485796d37ff4d76bc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:39.814062  6446 tablet_service.cc:1467] Processing CreateTablet for tablet 2cefdfa8851147999c1425c606f36b1f (DEFAULT_TABLE table=rebalance_test_table_9 [id=f406d4a60a5e40c485796d37ff4d76bc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:39.814555  6446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cefdfa8851147999c1425c606f36b1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:39.814555  6315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cefdfa8851147999c1425c606f36b1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:39.814469  6314 tablet_service.cc:1467] Processing CreateTablet for tablet f309b40931dd444091869119eab33bbc (DEFAULT_TABLE table=rebalance_test_table_9 [id=f406d4a60a5e40c485796d37ff4d76bc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:39.814942  6314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f309b40931dd444091869119eab33bbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:39.815354  6180 tablet_service.cc:1467] Processing CreateTablet for tablet f309b40931dd444091869119eab33bbc (DEFAULT_TABLE table=rebalance_test_table_9 [id=f406d4a60a5e40c485796d37ff4d76bc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:39.815799  6180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f309b40931dd444091869119eab33bbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:39.817456  6181 tablet_service.cc:1467] Processing CreateTablet for tablet 5be45a68cff24a75a03b2fcd4b7851f9 (DEFAULT_TABLE table=rebalance_test_table_9 [id=f406d4a60a5e40c485796d37ff4d76bc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:39.817899  6181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5be45a68cff24a75a03b2fcd4b7851f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:39.820715  6046 tablet_service.cc:1467] Processing CreateTablet for tablet 5be45a68cff24a75a03b2fcd4b7851f9 (DEFAULT_TABLE table=rebalance_test_table_9 [id=f406d4a60a5e40c485796d37ff4d76bc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:39.820837  6182 tablet_service.cc:1467] Processing CreateTablet for tablet 2cefdfa8851147999c1425c606f36b1f (DEFAULT_TABLE table=rebalance_test_table_9 [id=f406d4a60a5e40c485796d37ff4d76bc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:39.821221  6046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5be45a68cff24a75a03b2fcd4b7851f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:39.821228  6182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cefdfa8851147999c1425c606f36b1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:39.824366  6316 tablet_service.cc:1467] Processing CreateTablet for tablet 5be45a68cff24a75a03b2fcd4b7851f9 (DEFAULT_TABLE table=rebalance_test_table_9 [id=f406d4a60a5e40c485796d37ff4d76bc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:39.824620  6447 tablet_service.cc:1467] Processing CreateTablet for tablet f309b40931dd444091869119eab33bbc (DEFAULT_TABLE table=rebalance_test_table_9 [id=f406d4a60a5e40c485796d37ff4d76bc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:39.824865  6316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5be45a68cff24a75a03b2fcd4b7851f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:39.825054  6447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f309b40931dd444091869119eab33bbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:39.834695  7835 tablet_bootstrap.cc:492] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3: Bootstrap starting.
I20240502 01:55:39.837378  7835 tablet_bootstrap.cc:654] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:39.839665  7836 tablet_bootstrap.cc:492] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399: Bootstrap starting.
I20240502 01:55:39.841455  7836 tablet_bootstrap.cc:654] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:39.843094  7834 tablet_bootstrap.cc:492] T f309b40931dd444091869119eab33bbc P a2c90ee440b44b708bccc2e2d5385cc0: Bootstrap starting.
I20240502 01:55:39.845438  7834 tablet_bootstrap.cc:654] T f309b40931dd444091869119eab33bbc P a2c90ee440b44b708bccc2e2d5385cc0: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:39.849413  7839 tablet_bootstrap.cc:492] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9: Bootstrap starting.
I20240502 01:55:39.854110  7834 tablet_bootstrap.cc:492] T f309b40931dd444091869119eab33bbc P a2c90ee440b44b708bccc2e2d5385cc0: No bootstrap required, opened a new log
I20240502 01:55:39.854401  7834 ts_tablet_manager.cc:1367] T f309b40931dd444091869119eab33bbc P a2c90ee440b44b708bccc2e2d5385cc0: Time spent bootstrapping tablet: real 0.011s	user 0.003s	sys 0.007s
I20240502 01:55:39.855443  7839 tablet_bootstrap.cc:654] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:39.855317  7834 raft_consensus.cc:348] T f309b40931dd444091869119eab33bbc P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:39.855538  7834 raft_consensus.cc:374] T f309b40931dd444091869119eab33bbc P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:39.855638  7834 raft_consensus.cc:729] T f309b40931dd444091869119eab33bbc P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c90ee440b44b708bccc2e2d5385cc0, State: Initialized, Role: FOLLOWER
I20240502 01:55:39.856014  7834 consensus_queue.cc:260] T f309b40931dd444091869119eab33bbc P a2c90ee440b44b708bccc2e2d5385cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:39.856789  7834 ts_tablet_manager.cc:1398] T f309b40931dd444091869119eab33bbc P a2c90ee440b44b708bccc2e2d5385cc0: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240502 01:55:39.857067  7836 tablet_bootstrap.cc:492] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399: No bootstrap required, opened a new log
I20240502 01:55:39.857326  7834 tablet_bootstrap.cc:492] T 5be45a68cff24a75a03b2fcd4b7851f9 P a2c90ee440b44b708bccc2e2d5385cc0: Bootstrap starting.
I20240502 01:55:39.857342  7836 ts_tablet_manager.cc:1367] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399: Time spent bootstrapping tablet: real 0.018s	user 0.005s	sys 0.010s
I20240502 01:55:39.858381  7836 raft_consensus.cc:348] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:39.858712  7836 raft_consensus.cc:374] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:39.858843  7836 raft_consensus.cc:729] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07cdb213c77c4ffda1f999b66b725399, State: Initialized, Role: FOLLOWER
I20240502 01:55:39.859201  7834 tablet_bootstrap.cc:654] T 5be45a68cff24a75a03b2fcd4b7851f9 P a2c90ee440b44b708bccc2e2d5385cc0: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:39.859189  7836 consensus_queue.cc:260] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:39.861034  7836 ts_tablet_manager.cc:1398] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:55:39.861418  7836 tablet_bootstrap.cc:492] T 2cefdfa8851147999c1425c606f36b1f P 07cdb213c77c4ffda1f999b66b725399: Bootstrap starting.
I20240502 01:55:39.864642  7836 tablet_bootstrap.cc:654] T 2cefdfa8851147999c1425c606f36b1f P 07cdb213c77c4ffda1f999b66b725399: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:39.866653  7835 tablet_bootstrap.cc:492] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3: No bootstrap required, opened a new log
I20240502 01:55:39.866945  7835 ts_tablet_manager.cc:1367] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3: Time spent bootstrapping tablet: real 0.032s	user 0.007s	sys 0.015s
I20240502 01:55:39.868017  7835 raft_consensus.cc:348] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:39.868379  7835 raft_consensus.cc:374] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:39.868525  7835 raft_consensus.cc:729] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94ffd7aaf85b4e17881241b8396fa2f3, State: Initialized, Role: FOLLOWER
I20240502 01:55:39.868844  7835 consensus_queue.cc:260] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:39.869714  7835 ts_tablet_manager.cc:1398] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:55:39.870105  7835 tablet_bootstrap.cc:492] T 5be45a68cff24a75a03b2fcd4b7851f9 P 94ffd7aaf85b4e17881241b8396fa2f3: Bootstrap starting.
I20240502 01:55:39.870435  7839 tablet_bootstrap.cc:492] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9: No bootstrap required, opened a new log
I20240502 01:55:39.870678  7839 ts_tablet_manager.cc:1367] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9: Time spent bootstrapping tablet: real 0.022s	user 0.007s	sys 0.000s
I20240502 01:55:39.872011  7835 tablet_bootstrap.cc:654] T 5be45a68cff24a75a03b2fcd4b7851f9 P 94ffd7aaf85b4e17881241b8396fa2f3: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:39.873150  7839 raft_consensus.cc:348] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 01:55:39.873459  7839 raft_consensus.cc:374] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:39.873613  7839 raft_consensus.cc:729] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218bc8c1f9ce46059898fc738179adf9, State: Initialized, Role: FOLLOWER
I20240502 01:55:39.873986  7839 consensus_queue.cc:260] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 01:55:39.874696  7839 ts_tablet_manager.cc:1398] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 01:55:39.876459  7836 tablet_bootstrap.cc:492] T 2cefdfa8851147999c1425c606f36b1f P 07cdb213c77c4ffda1f999b66b725399: No bootstrap required, opened a new log
I20240502 01:55:39.876549  7834 tablet_bootstrap.cc:492] T 5be45a68cff24a75a03b2fcd4b7851f9 P a2c90ee440b44b708bccc2e2d5385cc0: No bootstrap required, opened a new log
I20240502 01:55:39.876678  7836 ts_tablet_manager.cc:1367] T 2cefdfa8851147999c1425c606f36b1f P 07cdb213c77c4ffda1f999b66b725399: Time spent bootstrapping tablet: real 0.015s	user 0.009s	sys 0.003s
I20240502 01:55:39.876727  7834 ts_tablet_manager.cc:1367] T 5be45a68cff24a75a03b2fcd4b7851f9 P a2c90ee440b44b708bccc2e2d5385cc0: Time spent bootstrapping tablet: real 0.020s	user 0.004s	sys 0.008s
I20240502 01:55:39.877519  7834 raft_consensus.cc:348] T 5be45a68cff24a75a03b2fcd4b7851f9 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 01:55:39.877517  7836 raft_consensus.cc:348] T 2cefdfa8851147999c1425c606f36b1f P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:39.877758  7834 raft_consensus.cc:374] T 5be45a68cff24a75a03b2fcd4b7851f9 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:39.877758  7836 raft_consensus.cc:374] T 2cefdfa8851147999c1425c606f36b1f P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:39.877825  7836 raft_consensus.cc:729] T 2cefdfa8851147999c1425c606f36b1f P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07cdb213c77c4ffda1f999b66b725399, State: Initialized, Role: FOLLOWER
I20240502 01:55:39.877832  7834 raft_consensus.cc:729] T 5be45a68cff24a75a03b2fcd4b7851f9 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c90ee440b44b708bccc2e2d5385cc0, State: Initialized, Role: FOLLOWER
I20240502 01:55:39.878093  7834 consensus_queue.cc:260] T 5be45a68cff24a75a03b2fcd4b7851f9 P a2c90ee440b44b708bccc2e2d5385cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 01:55:39.878156  7836 consensus_queue.cc:260] T 2cefdfa8851147999c1425c606f36b1f P 07cdb213c77c4ffda1f999b66b725399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:39.879413  7834 ts_tablet_manager.cc:1398] T 5be45a68cff24a75a03b2fcd4b7851f9 P a2c90ee440b44b708bccc2e2d5385cc0: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.002s
I20240502 01:55:39.879736  7834 tablet_bootstrap.cc:492] T 2cefdfa8851147999c1425c606f36b1f P a2c90ee440b44b708bccc2e2d5385cc0: Bootstrap starting.
I20240502 01:55:39.881042  7836 ts_tablet_manager.cc:1398] T 2cefdfa8851147999c1425c606f36b1f P 07cdb213c77c4ffda1f999b66b725399: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 01:55:39.881501  7834 tablet_bootstrap.cc:654] T 2cefdfa8851147999c1425c606f36b1f P a2c90ee440b44b708bccc2e2d5385cc0: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:39.885121  7835 tablet_bootstrap.cc:492] T 5be45a68cff24a75a03b2fcd4b7851f9 P 94ffd7aaf85b4e17881241b8396fa2f3: No bootstrap required, opened a new log
I20240502 01:55:39.885308  7835 ts_tablet_manager.cc:1367] T 5be45a68cff24a75a03b2fcd4b7851f9 P 94ffd7aaf85b4e17881241b8396fa2f3: Time spent bootstrapping tablet: real 0.015s	user 0.000s	sys 0.005s
I20240502 01:55:39.886063  7835 raft_consensus.cc:348] T 5be45a68cff24a75a03b2fcd4b7851f9 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 01:55:39.886265  7835 raft_consensus.cc:374] T 5be45a68cff24a75a03b2fcd4b7851f9 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:39.886336  7835 raft_consensus.cc:729] T 5be45a68cff24a75a03b2fcd4b7851f9 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94ffd7aaf85b4e17881241b8396fa2f3, State: Initialized, Role: FOLLOWER
I20240502 01:55:39.886574  7835 consensus_queue.cc:260] T 5be45a68cff24a75a03b2fcd4b7851f9 P 94ffd7aaf85b4e17881241b8396fa2f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 01:55:39.887032  7835 ts_tablet_manager.cc:1398] T 5be45a68cff24a75a03b2fcd4b7851f9 P 94ffd7aaf85b4e17881241b8396fa2f3: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:55:39.887477  7835 tablet_bootstrap.cc:492] T f309b40931dd444091869119eab33bbc P 94ffd7aaf85b4e17881241b8396fa2f3: Bootstrap starting.
I20240502 01:55:39.887701  7834 tablet_bootstrap.cc:492] T 2cefdfa8851147999c1425c606f36b1f P a2c90ee440b44b708bccc2e2d5385cc0: No bootstrap required, opened a new log
I20240502 01:55:39.887926  7834 ts_tablet_manager.cc:1367] T 2cefdfa8851147999c1425c606f36b1f P a2c90ee440b44b708bccc2e2d5385cc0: Time spent bootstrapping tablet: real 0.008s	user 0.003s	sys 0.001s
I20240502 01:55:39.888573  7834 raft_consensus.cc:348] T 2cefdfa8851147999c1425c606f36b1f P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:39.888744  7834 raft_consensus.cc:374] T 2cefdfa8851147999c1425c606f36b1f P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:39.888832  7834 raft_consensus.cc:729] T 2cefdfa8851147999c1425c606f36b1f P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c90ee440b44b708bccc2e2d5385cc0, State: Initialized, Role: FOLLOWER
I20240502 01:55:39.889077  7695 raft_consensus.cc:482] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:39.889101  7834 consensus_queue.cc:260] T 2cefdfa8851147999c1425c606f36b1f P a2c90ee440b44b708bccc2e2d5385cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:39.889240  7835 tablet_bootstrap.cc:654] T f309b40931dd444091869119eab33bbc P 94ffd7aaf85b4e17881241b8396fa2f3: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:39.889242  7695 raft_consensus.cc:504] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:39.889667  7834 ts_tablet_manager.cc:1398] T 2cefdfa8851147999c1425c606f36b1f P a2c90ee440b44b708bccc2e2d5385cc0: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240502 01:55:39.890072  7695 leader_election.cc:288] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197), 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867)
I20240502 01:55:39.891153  6203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cefdfa8851147999c1425c606f36b1f" candidate_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" is_pre_election: true
I20240502 01:55:39.891383  6203 raft_consensus.cc:2454] T 2cefdfa8851147999c1425c606f36b1f P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94ffd7aaf85b4e17881241b8396fa2f3 in term 0.
I20240502 01:55:39.892297  6469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cefdfa8851147999c1425c606f36b1f" candidate_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07cdb213c77c4ffda1f999b66b725399" is_pre_election: true
I20240502 01:55:39.892601  6469 raft_consensus.cc:2454] T 2cefdfa8851147999c1425c606f36b1f P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94ffd7aaf85b4e17881241b8396fa2f3 in term 0.
I20240502 01:55:39.893343  6271 leader_election.cc:302] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 07cdb213c77c4ffda1f999b66b725399, 94ffd7aaf85b4e17881241b8396fa2f3, a2c90ee440b44b708bccc2e2d5385cc0; no voters: 
I20240502 01:55:39.893793  7695 raft_consensus.cc:2781] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:39.893923  7695 raft_consensus.cc:482] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:39.894021  7695 raft_consensus.cc:3037] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:39.896905  7695 raft_consensus.cc:504] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:39.897747  7695 leader_election.cc:288] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 1 election: Requested vote from peers a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197), 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867)
I20240502 01:55:39.898353  6203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cefdfa8851147999c1425c606f36b1f" candidate_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c90ee440b44b708bccc2e2d5385cc0"
I20240502 01:55:39.898613  6203 raft_consensus.cc:3037] T 2cefdfa8851147999c1425c606f36b1f P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:39.900882  6467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cefdfa8851147999c1425c606f36b1f" candidate_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07cdb213c77c4ffda1f999b66b725399"
I20240502 01:55:39.901129  6467 raft_consensus.cc:3037] T 2cefdfa8851147999c1425c606f36b1f P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:39.901347  6203 raft_consensus.cc:2454] T 2cefdfa8851147999c1425c606f36b1f P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ffd7aaf85b4e17881241b8396fa2f3 in term 1.
I20240502 01:55:39.901547  7835 tablet_bootstrap.cc:492] T f309b40931dd444091869119eab33bbc P 94ffd7aaf85b4e17881241b8396fa2f3: No bootstrap required, opened a new log
I20240502 01:55:39.901715  7835 ts_tablet_manager.cc:1367] T f309b40931dd444091869119eab33bbc P 94ffd7aaf85b4e17881241b8396fa2f3: Time spent bootstrapping tablet: real 0.014s	user 0.008s	sys 0.004s
I20240502 01:55:39.901837  6271 leader_election.cc:302] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94ffd7aaf85b4e17881241b8396fa2f3, a2c90ee440b44b708bccc2e2d5385cc0; no voters: 
I20240502 01:55:39.902431  7769 raft_consensus.cc:2781] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:39.902465  7835 raft_consensus.cc:348] T f309b40931dd444091869119eab33bbc P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:39.902587  7769 raft_consensus.cc:686] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Becoming Leader. State: Replica: 94ffd7aaf85b4e17881241b8396fa2f3, State: Running, Role: LEADER
I20240502 01:55:39.902709  7835 raft_consensus.cc:374] T f309b40931dd444091869119eab33bbc P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:39.902814  7835 raft_consensus.cc:729] T f309b40931dd444091869119eab33bbc P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94ffd7aaf85b4e17881241b8396fa2f3, State: Initialized, Role: FOLLOWER
I20240502 01:55:39.902961  7769 consensus_queue.cc:237] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:39.903069  7835 consensus_queue.cc:260] T f309b40931dd444091869119eab33bbc P 94ffd7aaf85b4e17881241b8396fa2f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:39.903924  6467 raft_consensus.cc:2454] T 2cefdfa8851147999c1425c606f36b1f P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ffd7aaf85b4e17881241b8396fa2f3 in term 1.
I20240502 01:55:39.909883  7835 ts_tablet_manager.cc:1398] T f309b40931dd444091869119eab33bbc P 94ffd7aaf85b4e17881241b8396fa2f3: Time spent starting tablet: real 0.008s	user 0.000s	sys 0.008s
I20240502 01:55:39.910653  5916 catalog_manager.cc:5484] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131). New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } }
I20240502 01:55:39.958325  7776 raft_consensus.cc:482] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:39.958524  7776 raft_consensus.cc:504] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:39.959240  7776 leader_election.cc:288] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197), 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245)
I20240502 01:55:39.959826  6336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f309b40931dd444091869119eab33bbc" candidate_uuid: "07cdb213c77c4ffda1f999b66b725399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" is_pre_election: true
I20240502 01:55:39.959940  6203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f309b40931dd444091869119eab33bbc" candidate_uuid: "07cdb213c77c4ffda1f999b66b725399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" is_pre_election: true
I20240502 01:55:39.960152  6336 raft_consensus.cc:2454] T f309b40931dd444091869119eab33bbc P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07cdb213c77c4ffda1f999b66b725399 in term 0.
I20240502 01:55:39.960206  6203 raft_consensus.cc:2454] T f309b40931dd444091869119eab33bbc P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07cdb213c77c4ffda1f999b66b725399 in term 0.
I20240502 01:55:39.960613  6403 leader_election.cc:302] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07cdb213c77c4ffda1f999b66b725399, a2c90ee440b44b708bccc2e2d5385cc0; no voters: 
I20240502 01:55:39.960917  7776 raft_consensus.cc:2781] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:39.961042  7776 raft_consensus.cc:482] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:39.961125  7776 raft_consensus.cc:3037] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:39.964715  7776 raft_consensus.cc:504] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:39.965411  7776 leader_election.cc:288] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 [CANDIDATE]: Term 1 election: Requested vote from peers a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197), 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245)
I20240502 01:55:39.965888  6203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f309b40931dd444091869119eab33bbc" candidate_uuid: "07cdb213c77c4ffda1f999b66b725399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c90ee440b44b708bccc2e2d5385cc0"
I20240502 01:55:39.965977  6336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f309b40931dd444091869119eab33bbc" candidate_uuid: "07cdb213c77c4ffda1f999b66b725399" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ffd7aaf85b4e17881241b8396fa2f3"
I20240502 01:55:39.966125  6203 raft_consensus.cc:3037] T f309b40931dd444091869119eab33bbc P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:39.966213  6336 raft_consensus.cc:3037] T f309b40931dd444091869119eab33bbc P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:39.968192  6336 raft_consensus.cc:2454] T f309b40931dd444091869119eab33bbc P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07cdb213c77c4ffda1f999b66b725399 in term 1.
I20240502 01:55:39.968194  6203 raft_consensus.cc:2454] T f309b40931dd444091869119eab33bbc P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07cdb213c77c4ffda1f999b66b725399 in term 1.
I20240502 01:55:39.968694  6402 leader_election.cc:302] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07cdb213c77c4ffda1f999b66b725399, 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 
I20240502 01:55:39.969070  7776 raft_consensus.cc:2781] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:39.969213  7776 raft_consensus.cc:686] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Becoming Leader. State: Replica: 07cdb213c77c4ffda1f999b66b725399, State: Running, Role: LEADER
I20240502 01:55:39.969552  7776 consensus_queue.cc:237] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 01:55:39.972870  5916 catalog_manager.cc:5484] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132). New cstate: current_term: 1 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } }
I20240502 01:55:40.018077  7818 raft_consensus.cc:482] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:40.018306  7818 raft_consensus.cc:504] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 01:55:40.019179  7818 leader_election.cc:288] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197), 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245)
I20240502 01:55:40.019918  6204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5be45a68cff24a75a03b2fcd4b7851f9" candidate_uuid: "218bc8c1f9ce46059898fc738179adf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" is_pre_election: true
I20240502 01:55:40.020205  6204 raft_consensus.cc:2454] T 5be45a68cff24a75a03b2fcd4b7851f9 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 218bc8c1f9ce46059898fc738179adf9 in term 0.
I20240502 01:55:40.020550  6336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5be45a68cff24a75a03b2fcd4b7851f9" candidate_uuid: "218bc8c1f9ce46059898fc738179adf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" is_pre_election: true
I20240502 01:55:40.020700  6001 leader_election.cc:302] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9, a2c90ee440b44b708bccc2e2d5385cc0; no voters: 
I20240502 01:55:40.020804  6336 raft_consensus.cc:2454] T 5be45a68cff24a75a03b2fcd4b7851f9 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 218bc8c1f9ce46059898fc738179adf9 in term 0.
I20240502 01:55:40.021016  7818 raft_consensus.cc:2781] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:40.021147  7818 raft_consensus.cc:482] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:40.021250  7818 raft_consensus.cc:3037] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:40.023206  7818 raft_consensus.cc:504] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 01:55:40.023836  7818 leader_election.cc:288] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 1 election: Requested vote from peers a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197), 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245)
I20240502 01:55:40.024451  6204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5be45a68cff24a75a03b2fcd4b7851f9" candidate_uuid: "218bc8c1f9ce46059898fc738179adf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c90ee440b44b708bccc2e2d5385cc0"
I20240502 01:55:40.024451  6336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5be45a68cff24a75a03b2fcd4b7851f9" candidate_uuid: "218bc8c1f9ce46059898fc738179adf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ffd7aaf85b4e17881241b8396fa2f3"
I20240502 01:55:40.024688  6336 raft_consensus.cc:3037] T 5be45a68cff24a75a03b2fcd4b7851f9 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:40.024684  6204 raft_consensus.cc:3037] T 5be45a68cff24a75a03b2fcd4b7851f9 P a2c90ee440b44b708bccc2e2d5385cc0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:40.027571  6204 raft_consensus.cc:2454] T 5be45a68cff24a75a03b2fcd4b7851f9 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 218bc8c1f9ce46059898fc738179adf9 in term 1.
I20240502 01:55:40.027571  6336 raft_consensus.cc:2454] T 5be45a68cff24a75a03b2fcd4b7851f9 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 218bc8c1f9ce46059898fc738179adf9 in term 1.
I20240502 01:55:40.028142  6001 leader_election.cc:302] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9, 94ffd7aaf85b4e17881241b8396fa2f3, a2c90ee440b44b708bccc2e2d5385cc0; no voters: 
I20240502 01:55:40.028558  7818 raft_consensus.cc:2781] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:40.028707  7818 raft_consensus.cc:686] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 [term 1 LEADER]: Becoming Leader. State: Replica: 218bc8c1f9ce46059898fc738179adf9, State: Running, Role: LEADER
I20240502 01:55:40.029002  7818 consensus_queue.cc:237] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 01:55:40.032339  5916 catalog_manager.cc:5484] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129). New cstate: current_term: 1 leader_uuid: "218bc8c1f9ce46059898fc738179adf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } }
I20240502 01:55:40.284298  7769 consensus_queue.cc:1035] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:40.292356  7769 consensus_queue.cc:1035] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:40.352003  7776 consensus_queue.cc:1035] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:40.356882  7863 consensus_queue.cc:1035] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:55:40.451563  7853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:40.451973  7853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:40.460253  7853 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240502 01:55:40.515947  7818 consensus_queue.cc:1035] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:40.523311  7883 consensus_queue.cc:1035] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 255.798 ms
  time per row: 0.127899 ms
I20240502 01:55:40.904759 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.133:43379
--local_ip_for_outbound_sockets=127.25.109.133
--tserver_master_addrs=127.25.109.190:37927
--webserver_port=39661
--webserver_interface=127.25.109.133
--builtin_ntp_servers=127.25.109.148:46095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 01:55:41.285763  7918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:41.286238  7918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:41.286397  7918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:41.295009  7918 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:55:41.295173  7918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:41.295331  7918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.133
I20240502 01:55:41.306485  7918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:46095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.133:43379
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.25.109.133
--webserver_port=39661
--tserver_master_addrs=127.25.109.190:37927
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.133
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:41.308245  7918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:41.310257  7918 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:41.323854  7924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:41.324441  7926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:41.325042  7923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:41.325345  7918 server_base.cc:1034] running on GCE node
I20240502 01:55:41.326030  7918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:41.327062  7918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:41.328286  7918 hybrid_clock.cc:648] HybridClock initialized: now 1714614941328214 us; error 63 us; skew 500 ppm
I20240502 01:55:41.328645  7918 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:41.331035  7918 webserver.cc:460] Webserver started at http://127.25.109.133:39661/ using document root <none> and password file <none>
I20240502 01:55:41.331818  7918 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:41.331993  7918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:41.337960  7918 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.002s
I20240502 01:55:41.341063  7932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:41.342419  7918 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
I20240502 01:55:41.342579  7918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "60298ffea1dd4c25a38765c2ec9ec9d5"
format_stamp: "Formatted at 2024-05-02 01:55:23 on dist-test-slave-kj4b"
I20240502 01:55:41.343457  7918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:41.368696  7918 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:55:41.369621  7918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:41.369940  7918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:41.371011  7918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:41.372822  7918 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:41.372901  7918 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:41.373034  7918 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:41.373088  7918 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:41.404188  7918 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.133:43379
I20240502 01:55:41.404232  8046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.133:43379 every 8 connection(s)
I20240502 01:55:41.405746  7918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:55:41.406363 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 7918
I20240502 01:55:41.407236 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.134:42517
--local_ip_for_outbound_sockets=127.25.109.134
--tserver_master_addrs=127.25.109.190:37927
--webserver_port=40249
--webserver_interface=127.25.109.134
--builtin_ntp_servers=127.25.109.148:46095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:55:41.429105  8047 heartbeater.cc:347] Connected to a master server at 127.25.109.190:37927
I20240502 01:55:41.429490  8047 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:41.430507  8047 heartbeater.cc:511] Master 127.25.109.190:37927 requested a full tablet report, sending...
I20240502 01:55:41.432464  5922 ts_manager.cc:194] Re-registered known tserver with Master: 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133:43379)
I20240502 01:55:41.433339  5922 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.133:58253
W20240502 01:55:41.803053  8051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:41.803367  8051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:41.803486  8051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:41.811790  8051 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:55:41.811975  8051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:41.812141  8051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.134
I20240502 01:55:41.822611  8051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:46095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.134:42517
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.25.109.134
--webserver_port=40249
--tserver_master_addrs=127.25.109.190:37927
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.134
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:41.824208  8051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:41.826035  8051 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:41.839926  8058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:41.840399  8060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:41.841001  8057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:41.841511  8051 server_base.cc:1034] running on GCE node
I20240502 01:55:41.842170  8051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:41.843302  8051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:41.844537  8051 hybrid_clock.cc:648] HybridClock initialized: now 1714614941844460 us; error 63 us; skew 500 ppm
I20240502 01:55:41.844924  8051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:41.847522  8051 webserver.cc:460] Webserver started at http://127.25.109.134:40249/ using document root <none> and password file <none>
I20240502 01:55:41.848443  8051 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:41.848598  8051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:41.854750  8051 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240502 01:55:41.858201  8066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:41.859782  8051 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:55:41.860033  8051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "9db17f84d37944098cbeb9e0d8456efb"
format_stamp: "Formatted at 2024-05-02 01:55:24 on dist-test-slave-kj4b"
I20240502 01:55:41.860944  8051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:41.881755  8051 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:55:41.882659  8051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:41.882951  8051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:41.883991  8051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:41.885679  8051 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:41.885761  8051 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:41.885860  8051 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:41.885903  8051 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:41.918962  8051 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.134:42517
I20240502 01:55:41.919060  8180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.134:42517 every 8 connection(s)
I20240502 01:55:41.920724  8051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 01:55:41.925123 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 8051
I20240502 01:55:41.925782 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.135:40543
--local_ip_for_outbound_sockets=127.25.109.135
--tserver_master_addrs=127.25.109.190:37927
--webserver_port=45803
--webserver_interface=127.25.109.135
--builtin_ntp_servers=127.25.109.148:46095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:55:41.940171  8182 heartbeater.cc:347] Connected to a master server at 127.25.109.190:37927
I20240502 01:55:41.940536  8182 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:41.941435  8182 heartbeater.cc:511] Master 127.25.109.190:37927 requested a full tablet report, sending...
I20240502 01:55:41.943081  5922 ts_manager.cc:194] Re-registered known tserver with Master: 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
I20240502 01:55:41.944078  5922 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.134:60631
W20240502 01:55:42.304805  8185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:42.305126  8185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:42.305253  8185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:42.313690  8185 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:55:42.313874  8185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:42.314074  8185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.135
I20240502 01:55:42.324237  8185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:46095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.135:40543
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.25.109.135
--webserver_port=45803
--tserver_master_addrs=127.25.109.190:37927
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.135
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:42.325678  8185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:42.327764  8185 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:42.341408  8193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:42.341975  8192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:42.342607  8195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:42.342839  8185 server_base.cc:1034] running on GCE node
I20240502 01:55:42.343587  8185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:42.344720  8185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:42.345937  8185 hybrid_clock.cc:648] HybridClock initialized: now 1714614942345876 us; error 56 us; skew 500 ppm
I20240502 01:55:42.346285  8185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:42.348912  8185 webserver.cc:460] Webserver started at http://127.25.109.135:45803/ using document root <none> and password file <none>
I20240502 01:55:42.349704  8185 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:42.349848  8185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:42.355690  8185 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240502 01:55:42.359017  8201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:42.360584  8185 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 01:55:42.360776  8185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "436822a397664c54b8d1b9eb3bc6cff4"
format_stamp: "Formatted at 2024-05-02 01:55:24 on dist-test-slave-kj4b"
I20240502 01:55:42.361696  8185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:42.386487  8185 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:55:42.387377  8185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:42.387666  8185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:42.388725  8185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:42.390367  8185 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:42.390452  8185 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:42.390588  8185 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:42.390642  8185 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:42.432575  8185 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.135:40543
I20240502 01:55:42.432618  8314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.135:40543 every 8 connection(s)
I20240502 01:55:42.434221  8185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 01:55:42.436719  8047 heartbeater.cc:503] Master 127.25.109.190:37927 was elected leader, sending a full tablet report...
I20240502 01:55:42.441030 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 8185
I20240502 01:55:42.441855 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.136:44359
--local_ip_for_outbound_sockets=127.25.109.136
--tserver_master_addrs=127.25.109.190:37927
--webserver_port=34789
--webserver_interface=127.25.109.136
--builtin_ntp_servers=127.25.109.148:46095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 01:55:42.450338  8315 heartbeater.cc:347] Connected to a master server at 127.25.109.190:37927
I20240502 01:55:42.450680  8315 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:42.451479  8315 heartbeater.cc:511] Master 127.25.109.190:37927 requested a full tablet report, sending...
I20240502 01:55:42.453019  5922 ts_manager.cc:194] Re-registered known tserver with Master: 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135:40543)
I20240502 01:55:42.454104  5922 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.135:44777
W20240502 01:55:42.820120  8319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:42.820469  8319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:42.820600  8319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:42.828477  8319 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 01:55:42.828635  8319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:42.828805  8319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.136
I20240502 01:55:42.838150  8319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:46095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.136:44359
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.25.109.136
--webserver_port=34789
--tserver_master_addrs=127.25.109.190:37927
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.136
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:42.839493  8319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:42.841192  8319 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:42.853746  8324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:42.854295  8325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:42.854872  8327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:42.855129  8319 server_base.cc:1034] running on GCE node
I20240502 01:55:42.855851  8319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:42.857141  8319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:42.858353  8319 hybrid_clock.cc:648] HybridClock initialized: now 1714614942858305 us; error 41 us; skew 500 ppm
I20240502 01:55:42.858675  8319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:42.861209  8319 webserver.cc:460] Webserver started at http://127.25.109.136:34789/ using document root <none> and password file <none>
I20240502 01:55:42.861989  8319 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:42.862128  8319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:42.867923  8319 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.000s
I20240502 01:55:42.872005  8334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:42.873411  8319 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.001s	sys 0.003s
I20240502 01:55:42.873584  8319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "6d34e50fb5dd4a1db58013d4863b8992"
format_stamp: "Formatted at 2024-05-02 01:55:25 on dist-test-slave-kj4b"
I20240502 01:55:42.874398  8319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:42.918758  8319 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:55:42.919595  8319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:42.919914  8319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:42.921406  8319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:42.923234  8319 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:42.923316  8319 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:42.923444  8319 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:42.923492  8319 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:55:42.948025  8182 heartbeater.cc:503] Master 127.25.109.190:37927 was elected leader, sending a full tablet report...
I20240502 01:55:42.954870  8319 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.136:44359
I20240502 01:55:42.954927  8450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.136:44359 every 8 connection(s)
I20240502 01:55:42.956355  8319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 01:55:42.961706 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 8319
I20240502 01:55:42.971362  8451 heartbeater.cc:347] Connected to a master server at 127.25.109.190:37927
I20240502 01:55:42.971683  8451 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:42.972543  8451 heartbeater.cc:511] Master 127.25.109.190:37927 requested a full tablet report, sending...
I20240502 01:55:42.974009  5922 ts_manager.cc:194] Re-registered known tserver with Master: 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359)
I20240502 01:55:42.975014  5922 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.136:52745
I20240502 01:55:43.456873  8315 heartbeater.cc:503] Master 127.25.109.190:37927 was elected leader, sending a full tablet report...
I20240502 01:55:43.978096  8451 heartbeater.cc:503] Master 127.25.109.190:37927 was elected leader, sending a full tablet report...
I20240502 01:55:44.323228  6046 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:55:44.334668  6046 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:55:44.359898  6447 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:55:44.364255  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:44.386132  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:44.388609  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:55:44.400312  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:44.416419  8385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:44.421665  6046 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:55:44.438477  6447 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:55:44.443935  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:44.446753  6447 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:55:44.469444  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:44.488713  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:44.525174  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:44.560262  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:55:44.586427  8385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:44.605520  8385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:44.616199  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:55:44.655373  6134 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.132:36867, user_credentials={real_user=slave}} blocked reactor thread for 91421.2us
I20240502 01:55:44.692734  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:44.698355  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:55:44.726328  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:44.741056  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:44.767922  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:45.245807  6447 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:55:45.286120  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:45.318706  6046 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:55:45.341118  6447 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:55:45.359447  6046 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:55:45.402330  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:55:45.440696  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:45.446225  8385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:55:45.448927  6270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.129:36227, user_credentials={real_user=slave}} blocked reactor thread for 50729us
I20240502 01:55:45.461409  8385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:45.477609  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:55:45.483850  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:45.488819  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:45.501174  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:45.504961  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:45.525206  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:45.528647  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:46.025961  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:46.039646  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:46.045665  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:46.093323  8385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:46.106797  6046 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:55:46.113361  6447 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:55:46.136065  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:55:46.138033  6046 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:55:46.138391  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:46.155617  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:46.157004  6447 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:55:46.162685  8385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:46.183056  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:46.251341  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:46.264456  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:46.270917  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:55:46.575477  6447 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:55:46.606379  6046 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:55:46.638456  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:46.728922  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:46.730763  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:55:46.742131  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:46.755179  8385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:46.761206  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:47.012492  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:47.029210  8385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:47.030174  6447 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:55:47.036931  6046 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:55:47.086570  6447 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:55:47.102273  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:47.109916  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:55:47.113330  6046 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:55:47.126353  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:47.126396  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:47.155452  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:47.217254  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:55:47.244055  8385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:47.251550  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:47.260345  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:47.270004  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:47.612619  6447 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:55:47.633479  6046 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:55:47.672719  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:47.703001  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:47.720651  8385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:47.723472  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:47.769806  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:55:47.871129  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:47.899271  6046 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:55:47.902546  6447 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:55:47.911295  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:47.943753  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:47.969369  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:47.979619  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:55:47.997295  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:48.012454  8385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:48.199929  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:48.203262  6046 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:55:48.213587  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:48.255856  6447 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:55:48.261284  8385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:48.323031  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:48.330359  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:55:48.361629  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:48.534330  6467 consensus_queue.cc:237] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:55:48.586585  6067 raft_consensus.cc:1261] T 0cecd1b14e474dee825fae040ce1b31e P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:48.589133  6336 raft_consensus.cc:1261] T 0cecd1b14e474dee825fae040ce1b31e P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:48.592095  7850 consensus_queue.cc:1035] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:48.597741  7850 consensus_queue.cc:1035] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:55:48.623266  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0cecd1b14e474dee825fae040ce1b31e. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:48.657770  8639 raft_consensus.cc:2932] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:48.665819  6068 raft_consensus.cc:2932] T 0cecd1b14e474dee825fae040ce1b31e P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:48.673626  5922 catalog_manager.cc:5484] T 0cecd1b14e474dee825fae040ce1b31e P 218bc8c1f9ce46059898fc738179adf9 reported cstate change: config changed from index -1 to 4, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New cstate: current_term: 1 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:48.684448  6336 raft_consensus.cc:2932] T 0cecd1b14e474dee825fae040ce1b31e P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:48.835255  6447 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:55:48.836074  6447 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:55:48.843000  6046 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:55:48.878511  8385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:48.893565  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:48.895390  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:48.898348  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:48.914850  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:48.920933  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:48.926692  6046 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:55:48.950104  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:48.963601  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:48.987134  8385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:49.016744  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:55:49.031011  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:55:49.054090  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:49.055455  9277 ts_tablet_manager.cc:902] T 0cecd1b14e474dee825fae040ce1b31e P 9db17f84d37944098cbeb9e0d8456efb: Initiating tablet copy from peer 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867)
I20240502 01:55:49.134641  9277 tablet_copy_client.cc:308] T 0cecd1b14e474dee825fae040ce1b31e P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.132:36867
I20240502 01:55:49.178675  6489 tablet_copy_service.cc:140] P 07cdb213c77c4ffda1f999b66b725399: Received BeginTabletCopySession request for tablet 0cecd1b14e474dee825fae040ce1b31e from peer 9db17f84d37944098cbeb9e0d8456efb ({username='slave'} at 127.25.109.134:56199)
I20240502 01:55:49.178969  6489 tablet_copy_service.cc:161] P 07cdb213c77c4ffda1f999b66b725399: Beginning new tablet copy session on tablet 0cecd1b14e474dee825fae040ce1b31e from peer 9db17f84d37944098cbeb9e0d8456efb at {username='slave'} at 127.25.109.134:56199: session id = 9db17f84d37944098cbeb9e0d8456efb-0cecd1b14e474dee825fae040ce1b31e
I20240502 01:55:49.182916  6489 tablet_copy_source_session.cc:205] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:49.203802  9277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cecd1b14e474dee825fae040ce1b31e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:49.252589  9277 tablet_copy_client.cc:783] T 0cecd1b14e474dee825fae040ce1b31e P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:49.253116  9277 tablet_copy_client.cc:647] T 0cecd1b14e474dee825fae040ce1b31e P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:49.256089  9277 tablet_copy_client.cc:523] T 0cecd1b14e474dee825fae040ce1b31e P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:49.260062  6469 consensus_queue.cc:237] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 01:55:49.263059  9277 tablet_bootstrap.cc:492] T 0cecd1b14e474dee825fae040ce1b31e P 9db17f84d37944098cbeb9e0d8456efb: Bootstrap starting.
I20240502 01:55:49.279920  6204 consensus_queue.cc:237] T e4d4a1bd80f44b668a2d504e47fb744a P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:55:49.288733  6336 raft_consensus.cc:1261] T 0cecd1b14e474dee825fae040ce1b31e P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 01:55:49.289934  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:49.290189  6067 raft_consensus.cc:1261] T 0cecd1b14e474dee825fae040ce1b31e P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:49.290376  9309 consensus_queue.cc:1035] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:49.290915  9217 consensus_queue.cc:1035] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:49.300804  9217 raft_consensus.cc:2932] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:55:49.314167  6334 raft_consensus.cc:2932] T 0cecd1b14e474dee825fae040ce1b31e P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:55:49.314499  6067 raft_consensus.cc:2932] T 0cecd1b14e474dee825fae040ce1b31e P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:55:49.319826  9277 log.cc:824] T 0cecd1b14e474dee825fae040ce1b31e P 9db17f84d37944098cbeb9e0d8456efb: Log is configured to *not* fsync() on all Append() calls
I20240502 01:55:49.320887  5916 catalog_manager.cc:5484] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 reported cstate change: config changed from index 4 to 5, VOTER 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133) added. New cstate: current_term: 1 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
W20240502 01:55:49.326460  6134 consensus_peers.cc:485] T e4d4a1bd80f44b668a2d504e47fb744a P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135:40543): Couldn't send request to peer 436822a397664c54b8d1b9eb3bc6cff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e4d4a1bd80f44b668a2d504e47fb744a. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:49.347267  6334 raft_consensus.cc:1261] T e4d4a1bd80f44b668a2d504e47fb744a P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Refusing update from remote peer a2c90ee440b44b708bccc2e2d5385cc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:49.348438  8627 consensus_queue.cc:1035] T e4d4a1bd80f44b668a2d504e47fb744a P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:49.349531  6469 raft_consensus.cc:1261] T e4d4a1bd80f44b668a2d504e47fb744a P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Refusing update from remote peer a2c90ee440b44b708bccc2e2d5385cc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:55:49.350775  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133:43379): Couldn't send request to peer 60298ffea1dd4c25a38765c2ec9ec9d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0cecd1b14e474dee825fae040ce1b31e. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:49.351531  8627 consensus_queue.cc:1035] T e4d4a1bd80f44b668a2d504e47fb744a P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:49.394634  6469 raft_consensus.cc:2932] T e4d4a1bd80f44b668a2d504e47fb744a P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:55:49.406397  5913 catalog_manager.cc:5484] T e4d4a1bd80f44b668a2d504e47fb744a P 07cdb213c77c4ffda1f999b66b725399 reported cstate change: config changed from index -1 to 4, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New cstate: current_term: 1 leader_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:55:49.400794  9337 raft_consensus.cc:2932] T e4d4a1bd80f44b668a2d504e47fb744a P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:55:49.427652  6336 raft_consensus.cc:2932] T e4d4a1bd80f44b668a2d504e47fb744a P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:55:49.449841  9277 tablet_bootstrap.cc:492] T 0cecd1b14e474dee825fae040ce1b31e P 9db17f84d37944098cbeb9e0d8456efb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:49.450656  9277 tablet_bootstrap.cc:492] T 0cecd1b14e474dee825fae040ce1b31e P 9db17f84d37944098cbeb9e0d8456efb: Bootstrap complete.
I20240502 01:55:49.451375  9277 ts_tablet_manager.cc:1367] T 0cecd1b14e474dee825fae040ce1b31e P 9db17f84d37944098cbeb9e0d8456efb: Time spent bootstrapping tablet: real 0.189s	user 0.069s	sys 0.031s
I20240502 01:55:49.460172  9277 raft_consensus.cc:348] T 0cecd1b14e474dee825fae040ce1b31e P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:55:49.460978  9277 raft_consensus.cc:729] T 0cecd1b14e474dee825fae040ce1b31e P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9db17f84d37944098cbeb9e0d8456efb, State: Initialized, Role: FOLLOWER
I20240502 01:55:49.461980  9277 consensus_queue.cc:260] T 0cecd1b14e474dee825fae040ce1b31e P 9db17f84d37944098cbeb9e0d8456efb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:55:49.465221  9277 ts_tablet_manager.cc:1398] T 0cecd1b14e474dee825fae040ce1b31e P 9db17f84d37944098cbeb9e0d8456efb: Time spent starting tablet: real 0.014s	user 0.011s	sys 0.001s
I20240502 01:55:49.466732  6489 tablet_copy_service.cc:342] P 07cdb213c77c4ffda1f999b66b725399: Request end of tablet copy session 9db17f84d37944098cbeb9e0d8456efb-0cecd1b14e474dee825fae040ce1b31e received from {username='slave'} at 127.25.109.134:56199
I20240502 01:55:49.466928  6489 tablet_copy_service.cc:433] P 07cdb213c77c4ffda1f999b66b725399: ending tablet copy session 9db17f84d37944098cbeb9e0d8456efb-0cecd1b14e474dee825fae040ce1b31e on tablet 0cecd1b14e474dee825fae040ce1b31e with peer 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:55:49.545948  6334 consensus_queue.cc:237] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:55:49.562731  6204 raft_consensus.cc:1261] T 9a5f486852554e2fa2d07f7b9d686ced P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:49.563608  6067 raft_consensus.cc:1261] T 9a5f486852554e2fa2d07f7b9d686ced P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:49.564401  9165 consensus_queue.cc:1035] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:49.565956  9165 consensus_queue.cc:1035] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:49.580880  9165 raft_consensus.cc:2932] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:49.601394  6067 raft_consensus.cc:2932] T 9a5f486852554e2fa2d07f7b9d686ced P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:49.602020  6204 raft_consensus.cc:2932] T 9a5f486852554e2fa2d07f7b9d686ced P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:49.610400  5913 catalog_manager.cc:5484] T 9a5f486852554e2fa2d07f7b9d686ced P 218bc8c1f9ce46059898fc738179adf9 reported cstate change: config changed from index -1 to 4, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
W20240502 01:55:49.627173  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9a5f486852554e2fa2d07f7b9d686ced. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:49.665923  6334 consensus_queue.cc:237] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:55:49.671123  6204 raft_consensus.cc:1261] T 89f7cf12d85442749e7d5b259ebca43f P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:49.671571  6067 raft_consensus.cc:1261] T 89f7cf12d85442749e7d5b259ebca43f P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:49.672155  9289 consensus_queue.cc:1035] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:49.672608  9389 consensus_queue.cc:1035] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:55:49.681393  6268 consensus_peers.cc:485] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135:40543): Couldn't send request to peer 436822a397664c54b8d1b9eb3bc6cff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 89f7cf12d85442749e7d5b259ebca43f. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:49.693025  6469 consensus_queue.cc:237] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 01:55:49.693715  9418 ts_tablet_manager.cc:902] T 0cecd1b14e474dee825fae040ce1b31e P 60298ffea1dd4c25a38765c2ec9ec9d5: Initiating tablet copy from peer 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867)
I20240502 01:55:49.700980  9389 raft_consensus.cc:2932] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:55:49.704738  6067 raft_consensus.cc:2932] T 89f7cf12d85442749e7d5b259ebca43f P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:55:49.707223  6204 raft_consensus.cc:2932] T 89f7cf12d85442749e7d5b259ebca43f P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:55:49.711030  5921 catalog_manager.cc:5484] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3 reported cstate change: config changed from index -1 to 4, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:55:49.737627  6204 raft_consensus.cc:1261] T 4486b413e5c14ec6953e4895ee5797cf P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:55:49.737751  6402 consensus_peers.cc:485] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133:43379): Couldn't send request to peer 60298ffea1dd4c25a38765c2ec9ec9d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4486b413e5c14ec6953e4895ee5797cf. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:49.738500  9217 consensus_queue.cc:1035] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:49.752934  9418 tablet_copy_client.cc:308] T 0cecd1b14e474dee825fae040ce1b31e P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.132:36867
I20240502 01:55:49.776661  6067 raft_consensus.cc:1261] T 4486b413e5c14ec6953e4895ee5797cf P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:49.777731  9217 consensus_queue.cc:1035] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:49.782778  6334 consensus_queue.cc:237] T ac3325a06d8841718158e1f5f181a398 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 01:55:49.783062  6489 tablet_copy_service.cc:140] P 07cdb213c77c4ffda1f999b66b725399: Received BeginTabletCopySession request for tablet 0cecd1b14e474dee825fae040ce1b31e from peer 60298ffea1dd4c25a38765c2ec9ec9d5 ({username='slave'} at 127.25.109.133:42089)
I20240502 01:55:49.782827  9217 raft_consensus.cc:2932] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:55:49.783257  6489 tablet_copy_service.cc:161] P 07cdb213c77c4ffda1f999b66b725399: Beginning new tablet copy session on tablet 0cecd1b14e474dee825fae040ce1b31e from peer 60298ffea1dd4c25a38765c2ec9ec9d5 at {username='slave'} at 127.25.109.133:42089: session id = 60298ffea1dd4c25a38765c2ec9ec9d5-0cecd1b14e474dee825fae040ce1b31e
I20240502 01:55:49.786437  6489 tablet_copy_source_session.cc:205] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:49.788232  9444 ts_tablet_manager.cc:902] T e4d4a1bd80f44b668a2d504e47fb744a P 436822a397664c54b8d1b9eb3bc6cff4: Initiating tablet copy from peer a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197)
I20240502 01:55:49.790233  5921 catalog_manager.cc:5484] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 reported cstate change: config changed from index -1 to 4, VOTER 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133) added. New cstate: current_term: 1 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:55:49.790406  6204 raft_consensus.cc:2932] T 4486b413e5c14ec6953e4895ee5797cf P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:55:49.809815  6067 raft_consensus.cc:2932] T 4486b413e5c14ec6953e4895ee5797cf P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:55:49.822580  9418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cecd1b14e474dee825fae040ce1b31e. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 01:55:49.827697  6270 consensus_peers.cc:485] T ac3325a06d8841718158e1f5f181a398 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133:43379): Couldn't send request to peer 60298ffea1dd4c25a38765c2ec9ec9d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ac3325a06d8841718158e1f5f181a398. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:49.828255  6204 raft_consensus.cc:1261] T ac3325a06d8841718158e1f5f181a398 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:49.828943  9289 consensus_queue.cc:1035] T ac3325a06d8841718158e1f5f181a398 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:49.829499  6469 raft_consensus.cc:1261] T ac3325a06d8841718158e1f5f181a398 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:49.830147  9389 consensus_queue.cc:1035] T ac3325a06d8841718158e1f5f181a398 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:49.829444  8135 raft_consensus.cc:2932] T 0cecd1b14e474dee825fae040ce1b31e P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:55:49.857340  9444 tablet_copy_client.cc:308] T e4d4a1bd80f44b668a2d504e47fb744a P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.130:38197
I20240502 01:55:49.872097  9389 raft_consensus.cc:2932] T ac3325a06d8841718158e1f5f181a398 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:55:49.876173  6201 raft_consensus.cc:2932] T ac3325a06d8841718158e1f5f181a398 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:55:49.881551  6469 raft_consensus.cc:2932] T ac3325a06d8841718158e1f5f181a398 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:55:49.882967  5916 catalog_manager.cc:5484] T ac3325a06d8841718158e1f5f181a398 P a2c90ee440b44b708bccc2e2d5385cc0 reported cstate change: config changed from index -1 to 4, VOTER 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133) added. New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:55:49.922149  9418 tablet_copy_client.cc:783] T 0cecd1b14e474dee825fae040ce1b31e P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:49.923480  9418 tablet_copy_client.cc:647] T 0cecd1b14e474dee825fae040ce1b31e P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:49.927007  9418 tablet_copy_client.cc:523] T 0cecd1b14e474dee825fae040ce1b31e P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:49.939005  6224 tablet_copy_service.cc:140] P a2c90ee440b44b708bccc2e2d5385cc0: Received BeginTabletCopySession request for tablet e4d4a1bd80f44b668a2d504e47fb744a from peer 436822a397664c54b8d1b9eb3bc6cff4 ({username='slave'} at 127.25.109.135:34595)
I20240502 01:55:49.939379  6224 tablet_copy_service.cc:161] P a2c90ee440b44b708bccc2e2d5385cc0: Beginning new tablet copy session on tablet e4d4a1bd80f44b668a2d504e47fb744a from peer 436822a397664c54b8d1b9eb3bc6cff4 at {username='slave'} at 127.25.109.135:34595: session id = 436822a397664c54b8d1b9eb3bc6cff4-e4d4a1bd80f44b668a2d504e47fb744a
I20240502 01:55:49.943010  6224 tablet_copy_source_session.cc:205] T e4d4a1bd80f44b668a2d504e47fb744a P a2c90ee440b44b708bccc2e2d5385cc0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:49.951035  9444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4d4a1bd80f44b668a2d504e47fb744a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:49.957895  9418 tablet_bootstrap.cc:492] T 0cecd1b14e474dee825fae040ce1b31e P 60298ffea1dd4c25a38765c2ec9ec9d5: Bootstrap starting.
I20240502 01:55:49.987555  9444 tablet_copy_client.cc:783] T e4d4a1bd80f44b668a2d504e47fb744a P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:49.988032  9444 tablet_copy_client.cc:647] T e4d4a1bd80f44b668a2d504e47fb744a P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:49.993289  9444 tablet_copy_client.cc:523] T e4d4a1bd80f44b668a2d504e47fb744a P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:50.007153  6447 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:55:50.015652  9444 tablet_bootstrap.cc:492] T e4d4a1bd80f44b668a2d504e47fb744a P 436822a397664c54b8d1b9eb3bc6cff4: Bootstrap starting.
I20240502 01:55:50.046762  6046 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:55:50.065955  9491 ts_tablet_manager.cc:902] T 9a5f486852554e2fa2d07f7b9d686ced P 6d34e50fb5dd4a1db58013d4863b8992: Initiating tablet copy from peer 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245)
I20240502 01:55:50.081976  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:50.097196  8385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:50.100378  9418 log.cc:824] T 0cecd1b14e474dee825fae040ce1b31e P 60298ffea1dd4c25a38765c2ec9ec9d5: Log is configured to *not* fsync() on all Append() calls
I20240502 01:55:50.111999  9506 ts_tablet_manager.cc:902] T 89f7cf12d85442749e7d5b259ebca43f P 436822a397664c54b8d1b9eb3bc6cff4: Initiating tablet copy from peer 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245)
I20240502 01:55:50.118036  6067 consensus_queue.cc:237] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:55:50.123281  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:50.128469  9444 log.cc:824] T e4d4a1bd80f44b668a2d504e47fb744a P 436822a397664c54b8d1b9eb3bc6cff4: Log is configured to *not* fsync() on all Append() calls
I20240502 01:55:50.179837  9506 tablet_copy_client.cc:308] T 89f7cf12d85442749e7d5b259ebca43f P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.131:44245
I20240502 01:55:50.190099  6469 raft_consensus.cc:1261] T 899963a8260a42c4b73fb80eda2ecaa9 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:50.190510  6357 tablet_copy_service.cc:140] P 94ffd7aaf85b4e17881241b8396fa2f3: Received BeginTabletCopySession request for tablet 89f7cf12d85442749e7d5b259ebca43f from peer 436822a397664c54b8d1b9eb3bc6cff4 ({username='slave'} at 127.25.109.135:37865)
I20240502 01:55:50.190842  6357 tablet_copy_service.cc:161] P 94ffd7aaf85b4e17881241b8396fa2f3: Beginning new tablet copy session on tablet 89f7cf12d85442749e7d5b259ebca43f from peer 436822a397664c54b8d1b9eb3bc6cff4 at {username='slave'} at 127.25.109.135:37865: session id = 436822a397664c54b8d1b9eb3bc6cff4-89f7cf12d85442749e7d5b259ebca43f
I20240502 01:55:50.195197  6357 tablet_copy_source_session.cc:205] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:50.196519  9376 consensus_queue.cc:1035] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:50.200043  9506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89f7cf12d85442749e7d5b259ebca43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:50.208357  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:55:50.213832  9506 tablet_copy_client.cc:783] T 89f7cf12d85442749e7d5b259ebca43f P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:50.214458  9506 tablet_copy_client.cc:647] T 89f7cf12d85442749e7d5b259ebca43f P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:50.217490  9506 tablet_copy_client.cc:523] T 89f7cf12d85442749e7d5b259ebca43f P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:50.221621  9506 tablet_bootstrap.cc:492] T 89f7cf12d85442749e7d5b259ebca43f P 436822a397664c54b8d1b9eb3bc6cff4: Bootstrap starting.
I20240502 01:55:50.226624  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:50.244921  6334 raft_consensus.cc:1261] T 899963a8260a42c4b73fb80eda2ecaa9 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:50.248587  9376 consensus_queue.cc:1035] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:50.283519  6469 consensus_queue.cc:237] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
W20240502 01:55:50.292548  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 899963a8260a42c4b73fb80eda2ecaa9. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:50.295037  8951 raft_consensus.cc:2932] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:50.306257  6334 raft_consensus.cc:1261] T 6984d638991442e3adaed1b24bee95e9 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:50.307056  9317 consensus_queue.cc:1035] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:50.323204  9418 tablet_bootstrap.cc:492] T 0cecd1b14e474dee825fae040ce1b31e P 60298ffea1dd4c25a38765c2ec9ec9d5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:50.324009  9418 tablet_bootstrap.cc:492] T 0cecd1b14e474dee825fae040ce1b31e P 60298ffea1dd4c25a38765c2ec9ec9d5: Bootstrap complete.
I20240502 01:55:50.324553  9418 ts_tablet_manager.cc:1367] T 0cecd1b14e474dee825fae040ce1b31e P 60298ffea1dd4c25a38765c2ec9ec9d5: Time spent bootstrapping tablet: real 0.367s	user 0.073s	sys 0.000s
I20240502 01:55:50.331437  9418 raft_consensus.cc:348] T 0cecd1b14e474dee825fae040ce1b31e P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 01:55:50.332036  9418 raft_consensus.cc:729] T 0cecd1b14e474dee825fae040ce1b31e P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60298ffea1dd4c25a38765c2ec9ec9d5, State: Initialized, Role: FOLLOWER
I20240502 01:55:50.332878  9418 consensus_queue.cc:260] T 0cecd1b14e474dee825fae040ce1b31e P 60298ffea1dd4c25a38765c2ec9ec9d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 01:55:50.334342  6334 raft_consensus.cc:2932] T 899963a8260a42c4b73fb80eda2ecaa9 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:50.337232  9444 tablet_bootstrap.cc:492] T e4d4a1bd80f44b668a2d504e47fb744a P 436822a397664c54b8d1b9eb3bc6cff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:50.341259  9418 ts_tablet_manager.cc:1398] T 0cecd1b14e474dee825fae040ce1b31e P 60298ffea1dd4c25a38765c2ec9ec9d5: Time spent starting tablet: real 0.017s	user 0.015s	sys 0.000s
I20240502 01:55:50.341125  5920 catalog_manager.cc:5484] T 899963a8260a42c4b73fb80eda2ecaa9 P 94ffd7aaf85b4e17881241b8396fa2f3 reported cstate change: config changed from index -1 to 4, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New cstate: current_term: 1 leader_uuid: "218bc8c1f9ce46059898fc738179adf9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:50.340474  9506 tablet_bootstrap.cc:492] T 89f7cf12d85442749e7d5b259ebca43f P 436822a397664c54b8d1b9eb3bc6cff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:50.344259  9506 tablet_bootstrap.cc:492] T 89f7cf12d85442749e7d5b259ebca43f P 436822a397664c54b8d1b9eb3bc6cff4: Bootstrap complete.
I20240502 01:55:50.344264  9444 tablet_bootstrap.cc:492] T e4d4a1bd80f44b668a2d504e47fb744a P 436822a397664c54b8d1b9eb3bc6cff4: Bootstrap complete.
I20240502 01:55:50.344892  9506 ts_tablet_manager.cc:1367] T 89f7cf12d85442749e7d5b259ebca43f P 436822a397664c54b8d1b9eb3bc6cff4: Time spent bootstrapping tablet: real 0.123s	user 0.051s	sys 0.015s
I20240502 01:55:50.344892  9444 ts_tablet_manager.cc:1367] T e4d4a1bd80f44b668a2d504e47fb744a P 436822a397664c54b8d1b9eb3bc6cff4: Time spent bootstrapping tablet: real 0.330s	user 0.068s	sys 0.006s
I20240502 01:55:50.345746  6489 tablet_copy_service.cc:342] P 07cdb213c77c4ffda1f999b66b725399: Request end of tablet copy session 60298ffea1dd4c25a38765c2ec9ec9d5-0cecd1b14e474dee825fae040ce1b31e received from {username='slave'} at 127.25.109.133:42089
I20240502 01:55:50.345901  6489 tablet_copy_service.cc:433] P 07cdb213c77c4ffda1f999b66b725399: ending tablet copy session 60298ffea1dd4c25a38765c2ec9ec9d5-0cecd1b14e474dee825fae040ce1b31e on tablet 0cecd1b14e474dee825fae040ce1b31e with peer 60298ffea1dd4c25a38765c2ec9ec9d5
I20240502 01:55:50.347404  9418 ts_tablet_manager.cc:902] T ac3325a06d8841718158e1f5f181a398 P 60298ffea1dd4c25a38765c2ec9ec9d5: Initiating tablet copy from peer 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245)
I20240502 01:55:50.358755  9444 raft_consensus.cc:348] T e4d4a1bd80f44b668a2d504e47fb744a P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:55:50.359613  9444 raft_consensus.cc:729] T e4d4a1bd80f44b668a2d504e47fb744a P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 436822a397664c54b8d1b9eb3bc6cff4, State: Initialized, Role: FOLLOWER
I20240502 01:55:50.358755  9506 raft_consensus.cc:348] T 89f7cf12d85442749e7d5b259ebca43f P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:55:50.360821  9444 consensus_queue.cc:260] T e4d4a1bd80f44b668a2d504e47fb744a P 436822a397664c54b8d1b9eb3bc6cff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:55:50.361449  9506 raft_consensus.cc:729] T 89f7cf12d85442749e7d5b259ebca43f P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 436822a397664c54b8d1b9eb3bc6cff4, State: Initialized, Role: FOLLOWER
I20240502 01:55:50.361495  6067 raft_consensus.cc:1261] T 6984d638991442e3adaed1b24bee95e9 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:50.362011  9506 consensus_queue.cc:260] T 89f7cf12d85442749e7d5b259ebca43f P 436822a397664c54b8d1b9eb3bc6cff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:55:50.362228  9317 consensus_queue.cc:1035] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:50.372207  9506 ts_tablet_manager.cc:1398] T 89f7cf12d85442749e7d5b259ebca43f P 436822a397664c54b8d1b9eb3bc6cff4: Time spent starting tablet: real 0.027s	user 0.008s	sys 0.004s
I20240502 01:55:50.374723  9444 ts_tablet_manager.cc:1398] T e4d4a1bd80f44b668a2d504e47fb744a P 436822a397664c54b8d1b9eb3bc6cff4: Time spent starting tablet: real 0.029s	user 0.009s	sys 0.000s
I20240502 01:55:50.375836  6224 tablet_copy_service.cc:342] P a2c90ee440b44b708bccc2e2d5385cc0: Request end of tablet copy session 436822a397664c54b8d1b9eb3bc6cff4-e4d4a1bd80f44b668a2d504e47fb744a received from {username='slave'} at 127.25.109.135:34595
I20240502 01:55:50.376092  6224 tablet_copy_service.cc:433] P a2c90ee440b44b708bccc2e2d5385cc0: ending tablet copy session 436822a397664c54b8d1b9eb3bc6cff4-e4d4a1bd80f44b668a2d504e47fb744a on tablet e4d4a1bd80f44b668a2d504e47fb744a with peer 436822a397664c54b8d1b9eb3bc6cff4
I20240502 01:55:50.378713  6357 tablet_copy_service.cc:342] P 94ffd7aaf85b4e17881241b8396fa2f3: Request end of tablet copy session 436822a397664c54b8d1b9eb3bc6cff4-89f7cf12d85442749e7d5b259ebca43f received from {username='slave'} at 127.25.109.135:37865
I20240502 01:55:50.378945  6357 tablet_copy_service.cc:433] P 94ffd7aaf85b4e17881241b8396fa2f3: ending tablet copy session 436822a397664c54b8d1b9eb3bc6cff4-89f7cf12d85442749e7d5b259ebca43f on tablet 89f7cf12d85442749e7d5b259ebca43f with peer 436822a397664c54b8d1b9eb3bc6cff4
W20240502 01:55:50.399156  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6984d638991442e3adaed1b24bee95e9. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:50.418363  9491 tablet_copy_client.cc:308] T 9a5f486852554e2fa2d07f7b9d686ced P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.131:44245
I20240502 01:55:50.433655  9531 ts_tablet_manager.cc:902] T 4486b413e5c14ec6953e4895ee5797cf P 60298ffea1dd4c25a38765c2ec9ec9d5: Initiating tablet copy from peer 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867)
I20240502 01:55:50.447741  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:50.473021  9317 raft_consensus.cc:2932] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:50.477376  6335 raft_consensus.cc:2932] T 6984d638991442e3adaed1b24bee95e9 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:50.484400  5920 catalog_manager.cc:5484] T 6984d638991442e3adaed1b24bee95e9 P 94ffd7aaf85b4e17881241b8396fa2f3 reported cstate change: config changed from index -1 to 4, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New cstate: current_term: 1 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:50.489419  6068 raft_consensus.cc:2932] T 6984d638991442e3adaed1b24bee95e9 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:50.505378  6469 raft_consensus.cc:2932] T 899963a8260a42c4b73fb80eda2ecaa9 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:50.508118  9418 tablet_copy_client.cc:308] T ac3325a06d8841718158e1f5f181a398 P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.131:44245
I20240502 01:55:50.509593  6357 tablet_copy_service.cc:140] P 94ffd7aaf85b4e17881241b8396fa2f3: Received BeginTabletCopySession request for tablet 9a5f486852554e2fa2d07f7b9d686ced from peer 6d34e50fb5dd4a1db58013d4863b8992 ({username='slave'} at 127.25.109.136:52593)
I20240502 01:55:50.509789  6357 tablet_copy_service.cc:161] P 94ffd7aaf85b4e17881241b8396fa2f3: Beginning new tablet copy session on tablet 9a5f486852554e2fa2d07f7b9d686ced from peer 6d34e50fb5dd4a1db58013d4863b8992 at {username='slave'} at 127.25.109.136:52593: session id = 6d34e50fb5dd4a1db58013d4863b8992-9a5f486852554e2fa2d07f7b9d686ced
I20240502 01:55:50.512987  6357 tablet_copy_source_session.cc:205] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:50.518921  9491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a5f486852554e2fa2d07f7b9d686ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:50.582885  6357 tablet_copy_service.cc:140] P 94ffd7aaf85b4e17881241b8396fa2f3: Received BeginTabletCopySession request for tablet ac3325a06d8841718158e1f5f181a398 from peer 60298ffea1dd4c25a38765c2ec9ec9d5 ({username='slave'} at 127.25.109.133:43975)
I20240502 01:55:50.583138  6357 tablet_copy_service.cc:161] P 94ffd7aaf85b4e17881241b8396fa2f3: Beginning new tablet copy session on tablet ac3325a06d8841718158e1f5f181a398 from peer 60298ffea1dd4c25a38765c2ec9ec9d5 at {username='slave'} at 127.25.109.133:43975: session id = 60298ffea1dd4c25a38765c2ec9ec9d5-ac3325a06d8841718158e1f5f181a398
I20240502 01:55:50.586575  6357 tablet_copy_source_session.cc:205] T ac3325a06d8841718158e1f5f181a398 P 94ffd7aaf85b4e17881241b8396fa2f3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:50.589231  9418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac3325a06d8841718158e1f5f181a398. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:50.595927  9418 tablet_copy_client.cc:783] T ac3325a06d8841718158e1f5f181a398 P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:50.596228  9418 tablet_copy_client.cc:647] T ac3325a06d8841718158e1f5f181a398 P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:50.598963  9418 tablet_copy_client.cc:523] T ac3325a06d8841718158e1f5f181a398 P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:50.608382  9531 tablet_copy_client.cc:308] T 4486b413e5c14ec6953e4895ee5797cf P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.132:36867
I20240502 01:55:50.609452  6489 tablet_copy_service.cc:140] P 07cdb213c77c4ffda1f999b66b725399: Received BeginTabletCopySession request for tablet 4486b413e5c14ec6953e4895ee5797cf from peer 60298ffea1dd4c25a38765c2ec9ec9d5 ({username='slave'} at 127.25.109.133:42089)
I20240502 01:55:50.609644  6489 tablet_copy_service.cc:161] P 07cdb213c77c4ffda1f999b66b725399: Beginning new tablet copy session on tablet 4486b413e5c14ec6953e4895ee5797cf from peer 60298ffea1dd4c25a38765c2ec9ec9d5 at {username='slave'} at 127.25.109.133:42089: session id = 60298ffea1dd4c25a38765c2ec9ec9d5-4486b413e5c14ec6953e4895ee5797cf
I20240502 01:55:50.610160  9418 tablet_bootstrap.cc:492] T ac3325a06d8841718158e1f5f181a398 P 60298ffea1dd4c25a38765c2ec9ec9d5: Bootstrap starting.
I20240502 01:55:50.612962  6489 tablet_copy_source_session.cc:205] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:50.614943  9531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4486b413e5c14ec6953e4895ee5797cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:50.644045  9491 tablet_copy_client.cc:783] T 9a5f486852554e2fa2d07f7b9d686ced P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:50.651122  8270 raft_consensus.cc:1203] T 89f7cf12d85442749e7d5b259ebca43f P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:55:50.658115  9491 tablet_copy_client.cc:647] T 9a5f486852554e2fa2d07f7b9d686ced P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:50.663111  9491 tablet_copy_client.cc:523] T 9a5f486852554e2fa2d07f7b9d686ced P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:50.664877  6204 consensus_queue.cc:237] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:55:50.685819  6334 raft_consensus.cc:1261] T d7922fd756344d14a336ff0c54afdf73 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Refusing update from remote peer a2c90ee440b44b708bccc2e2d5385cc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:50.686539  9337 consensus_queue.cc:1035] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:50.686828  9491 tablet_bootstrap.cc:492] T 9a5f486852554e2fa2d07f7b9d686ced P 6d34e50fb5dd4a1db58013d4863b8992: Bootstrap starting.
I20240502 01:55:50.697213  6469 raft_consensus.cc:1261] T d7922fd756344d14a336ff0c54afdf73 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Refusing update from remote peer a2c90ee440b44b708bccc2e2d5385cc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:50.699952  9337 consensus_queue.cc:1035] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:50.727763  9337 raft_consensus.cc:2932] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:50.742460  9598 ts_tablet_manager.cc:902] T 899963a8260a42c4b73fb80eda2ecaa9 P 9db17f84d37944098cbeb9e0d8456efb: Initiating tablet copy from peer 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227)
I20240502 01:55:50.745409  9531 tablet_copy_client.cc:783] T 4486b413e5c14ec6953e4895ee5797cf P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:50.746196  9491 log.cc:824] T 9a5f486852554e2fa2d07f7b9d686ced P 6d34e50fb5dd4a1db58013d4863b8992: Log is configured to *not* fsync() on all Append() calls
I20240502 01:55:50.748652  9531 tablet_copy_client.cc:647] T 4486b413e5c14ec6953e4895ee5797cf P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:50.751163  6334 raft_consensus.cc:2932] T d7922fd756344d14a336ff0c54afdf73 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:50.757637  5920 catalog_manager.cc:5484] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 reported cstate change: config changed from index -1 to 4, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New cstate: current_term: 1 leader_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:50.757558  6469 raft_consensus.cc:2932] T d7922fd756344d14a336ff0c54afdf73 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:50.766745  9531 tablet_copy_client.cc:523] T 4486b413e5c14ec6953e4895ee5797cf P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240502 01:55:50.767390  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7922fd756344d14a336ff0c54afdf73. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:50.777098  8001 raft_consensus.cc:1203] T 0cecd1b14e474dee825fae040ce1b31e P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 01:55:50.791167  8270 raft_consensus.cc:1203] T e4d4a1bd80f44b668a2d504e47fb744a P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:55:50.800093  9418 tablet_bootstrap.cc:492] T ac3325a06d8841718158e1f5f181a398 P 60298ffea1dd4c25a38765c2ec9ec9d5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:50.800712  9418 tablet_bootstrap.cc:492] T ac3325a06d8841718158e1f5f181a398 P 60298ffea1dd4c25a38765c2ec9ec9d5: Bootstrap complete.
I20240502 01:55:50.801081  9418 ts_tablet_manager.cc:1367] T ac3325a06d8841718158e1f5f181a398 P 60298ffea1dd4c25a38765c2ec9ec9d5: Time spent bootstrapping tablet: real 0.191s	user 0.046s	sys 0.005s
I20240502 01:55:50.802186  9418 raft_consensus.cc:348] T ac3325a06d8841718158e1f5f181a398 P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 01:55:50.802548  9418 raft_consensus.cc:729] T ac3325a06d8841718158e1f5f181a398 P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60298ffea1dd4c25a38765c2ec9ec9d5, State: Initialized, Role: FOLLOWER
I20240502 01:55:50.802886  9418 consensus_queue.cc:260] T ac3325a06d8841718158e1f5f181a398 P 60298ffea1dd4c25a38765c2ec9ec9d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 01:55:50.803208  9531 tablet_bootstrap.cc:492] T 4486b413e5c14ec6953e4895ee5797cf P 60298ffea1dd4c25a38765c2ec9ec9d5: Bootstrap starting.
I20240502 01:55:50.804309  9418 ts_tablet_manager.cc:1398] T ac3325a06d8841718158e1f5f181a398 P 60298ffea1dd4c25a38765c2ec9ec9d5: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240502 01:55:50.809512  9598 tablet_copy_client.cc:308] T 899963a8260a42c4b73fb80eda2ecaa9 P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.129:36227
I20240502 01:55:50.809732  6357 tablet_copy_service.cc:342] P 94ffd7aaf85b4e17881241b8396fa2f3: Request end of tablet copy session 60298ffea1dd4c25a38765c2ec9ec9d5-ac3325a06d8841718158e1f5f181a398 received from {username='slave'} at 127.25.109.133:43975
I20240502 01:55:50.809919  6357 tablet_copy_service.cc:433] P 94ffd7aaf85b4e17881241b8396fa2f3: ending tablet copy session 60298ffea1dd4c25a38765c2ec9ec9d5-ac3325a06d8841718158e1f5f181a398 on tablet ac3325a06d8841718158e1f5f181a398 with peer 60298ffea1dd4c25a38765c2ec9ec9d5
I20240502 01:55:50.825134  9491 tablet_bootstrap.cc:492] T 9a5f486852554e2fa2d07f7b9d686ced P 6d34e50fb5dd4a1db58013d4863b8992: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:50.825811  9491 tablet_bootstrap.cc:492] T 9a5f486852554e2fa2d07f7b9d686ced P 6d34e50fb5dd4a1db58013d4863b8992: Bootstrap complete.
I20240502 01:55:50.826370  9491 ts_tablet_manager.cc:1367] T 9a5f486852554e2fa2d07f7b9d686ced P 6d34e50fb5dd4a1db58013d4863b8992: Time spent bootstrapping tablet: real 0.140s	user 0.061s	sys 0.007s
I20240502 01:55:50.831255  6068 consensus_queue.cc:237] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:55:50.832957  9491 raft_consensus.cc:348] T 9a5f486852554e2fa2d07f7b9d686ced P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:55:50.833467  9491 raft_consensus.cc:729] T 9a5f486852554e2fa2d07f7b9d686ced P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d34e50fb5dd4a1db58013d4863b8992, State: Initialized, Role: FOLLOWER
I20240502 01:55:50.834363  9491 consensus_queue.cc:260] T 9a5f486852554e2fa2d07f7b9d686ced P 6d34e50fb5dd4a1db58013d4863b8992 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:55:50.878873  6201 raft_consensus.cc:1261] T 99e74a49b42745888058db67049dc682 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:50.879832  8951 consensus_queue.cc:1035] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:50.904371  6469 raft_consensus.cc:1261] T 99e74a49b42745888058db67049dc682 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:50.906035  8951 consensus_queue.cc:1035] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:50.910825  9636 ts_tablet_manager.cc:902] T 6984d638991442e3adaed1b24bee95e9 P 6d34e50fb5dd4a1db58013d4863b8992: Initiating tablet copy from peer 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867)
I20240502 01:55:50.918154  6089 tablet_copy_service.cc:140] P 218bc8c1f9ce46059898fc738179adf9: Received BeginTabletCopySession request for tablet 899963a8260a42c4b73fb80eda2ecaa9 from peer 9db17f84d37944098cbeb9e0d8456efb ({username='slave'} at 127.25.109.134:39709)
I20240502 01:55:50.918473  6089 tablet_copy_service.cc:161] P 218bc8c1f9ce46059898fc738179adf9: Beginning new tablet copy session on tablet 899963a8260a42c4b73fb80eda2ecaa9 from peer 9db17f84d37944098cbeb9e0d8456efb at {username='slave'} at 127.25.109.134:39709: session id = 9db17f84d37944098cbeb9e0d8456efb-899963a8260a42c4b73fb80eda2ecaa9
I20240502 01:55:50.922708  6089 tablet_copy_source_session.cc:205] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:50.926283  9531 tablet_bootstrap.cc:492] T 4486b413e5c14ec6953e4895ee5797cf P 60298ffea1dd4c25a38765c2ec9ec9d5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:50.926887  9531 tablet_bootstrap.cc:492] T 4486b413e5c14ec6953e4895ee5797cf P 60298ffea1dd4c25a38765c2ec9ec9d5: Bootstrap complete.
I20240502 01:55:50.926939  9598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 899963a8260a42c4b73fb80eda2ecaa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:50.927227  9531 ts_tablet_manager.cc:1367] T 4486b413e5c14ec6953e4895ee5797cf P 60298ffea1dd4c25a38765c2ec9ec9d5: Time spent bootstrapping tablet: real 0.124s	user 0.047s	sys 0.009s
I20240502 01:55:50.928144  9531 raft_consensus.cc:348] T 4486b413e5c14ec6953e4895ee5797cf P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 01:55:50.928391  9531 raft_consensus.cc:729] T 4486b413e5c14ec6953e4895ee5797cf P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60298ffea1dd4c25a38765c2ec9ec9d5, State: Initialized, Role: FOLLOWER
I20240502 01:55:50.928707  9531 consensus_queue.cc:260] T 4486b413e5c14ec6953e4895ee5797cf P 60298ffea1dd4c25a38765c2ec9ec9d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 01:55:50.929484  9531 ts_tablet_manager.cc:1398] T 4486b413e5c14ec6953e4895ee5797cf P 60298ffea1dd4c25a38765c2ec9ec9d5: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:55:50.930989  6489 tablet_copy_service.cc:342] P 07cdb213c77c4ffda1f999b66b725399: Request end of tablet copy session 60298ffea1dd4c25a38765c2ec9ec9d5-4486b413e5c14ec6953e4895ee5797cf received from {username='slave'} at 127.25.109.133:42089
I20240502 01:55:50.931138  6489 tablet_copy_service.cc:433] P 07cdb213c77c4ffda1f999b66b725399: ending tablet copy session 60298ffea1dd4c25a38765c2ec9ec9d5-4486b413e5c14ec6953e4895ee5797cf on tablet 4486b413e5c14ec6953e4895ee5797cf with peer 60298ffea1dd4c25a38765c2ec9ec9d5
I20240502 01:55:50.932948  8951 raft_consensus.cc:2932] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:55:50.955549  9598 tablet_copy_client.cc:783] T 899963a8260a42c4b73fb80eda2ecaa9 P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:50.956218  9598 tablet_copy_client.cc:647] T 899963a8260a42c4b73fb80eda2ecaa9 P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:50.971436  9598 tablet_copy_client.cc:523] T 899963a8260a42c4b73fb80eda2ecaa9 P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:50.972123  6201 raft_consensus.cc:2932] T 99e74a49b42745888058db67049dc682 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:55:50.985474  9491 ts_tablet_manager.cc:1398] T 9a5f486852554e2fa2d07f7b9d686ced P 6d34e50fb5dd4a1db58013d4863b8992: Time spent starting tablet: real 0.159s	user 0.030s	sys 0.069s
I20240502 01:55:50.986330  6357 tablet_copy_service.cc:342] P 94ffd7aaf85b4e17881241b8396fa2f3: Request end of tablet copy session 6d34e50fb5dd4a1db58013d4863b8992-9a5f486852554e2fa2d07f7b9d686ced received from {username='slave'} at 127.25.109.136:52593
I20240502 01:55:50.986505  6357 tablet_copy_service.cc:433] P 94ffd7aaf85b4e17881241b8396fa2f3: ending tablet copy session 6d34e50fb5dd4a1db58013d4863b8992-9a5f486852554e2fa2d07f7b9d686ced on tablet 9a5f486852554e2fa2d07f7b9d686ced with peer 6d34e50fb5dd4a1db58013d4863b8992
I20240502 01:55:50.987537  6469 raft_consensus.cc:2932] T 99e74a49b42745888058db67049dc682 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:55:50.996476  9636 tablet_copy_client.cc:308] T 6984d638991442e3adaed1b24bee95e9 P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.132:36867
I20240502 01:55:50.996690  5916 catalog_manager.cc:5484] T 99e74a49b42745888058db67049dc682 P 07cdb213c77c4ffda1f999b66b725399 reported cstate change: config changed from index -1 to 4, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New cstate: current_term: 1 leader_uuid: "218bc8c1f9ce46059898fc738179adf9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
W20240502 01:55:51.019953  5998 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135:40543): Couldn't send request to peer 436822a397664c54b8d1b9eb3bc6cff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99e74a49b42745888058db67049dc682. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:51.020488  9598 tablet_bootstrap.cc:492] T 899963a8260a42c4b73fb80eda2ecaa9 P 9db17f84d37944098cbeb9e0d8456efb: Bootstrap starting.
I20240502 01:55:51.060593  6489 tablet_copy_service.cc:140] P 07cdb213c77c4ffda1f999b66b725399: Received BeginTabletCopySession request for tablet 6984d638991442e3adaed1b24bee95e9 from peer 6d34e50fb5dd4a1db58013d4863b8992 ({username='slave'} at 127.25.109.136:34855)
I20240502 01:55:51.060832  6489 tablet_copy_service.cc:161] P 07cdb213c77c4ffda1f999b66b725399: Beginning new tablet copy session on tablet 6984d638991442e3adaed1b24bee95e9 from peer 6d34e50fb5dd4a1db58013d4863b8992 at {username='slave'} at 127.25.109.136:34855: session id = 6d34e50fb5dd4a1db58013d4863b8992-6984d638991442e3adaed1b24bee95e9
I20240502 01:55:51.063745  6489 tablet_copy_source_session.cc:205] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:51.065680  9636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6984d638991442e3adaed1b24bee95e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:51.072460  9636 tablet_copy_client.cc:783] T 6984d638991442e3adaed1b24bee95e9 P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:51.072878  9636 tablet_copy_client.cc:647] T 6984d638991442e3adaed1b24bee95e9 P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:51.074985  9636 tablet_copy_client.cc:523] T 6984d638991442e3adaed1b24bee95e9 P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:51.075232  8406 raft_consensus.cc:1203] T 9a5f486852554e2fa2d07f7b9d686ced P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:55:51.081676  9636 tablet_bootstrap.cc:492] T 6984d638991442e3adaed1b24bee95e9 P 6d34e50fb5dd4a1db58013d4863b8992: Bootstrap starting.
I20240502 01:55:51.112062  9491 ts_tablet_manager.cc:902] T d7922fd756344d14a336ff0c54afdf73 P 6d34e50fb5dd4a1db58013d4863b8992: Initiating tablet copy from peer a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197)
I20240502 01:55:51.127079  9491 tablet_copy_client.cc:308] T d7922fd756344d14a336ff0c54afdf73 P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.130:38197
I20240502 01:55:51.137511  6224 tablet_copy_service.cc:140] P a2c90ee440b44b708bccc2e2d5385cc0: Received BeginTabletCopySession request for tablet d7922fd756344d14a336ff0c54afdf73 from peer 6d34e50fb5dd4a1db58013d4863b8992 ({username='slave'} at 127.25.109.136:34777)
I20240502 01:55:51.137738  6224 tablet_copy_service.cc:161] P a2c90ee440b44b708bccc2e2d5385cc0: Beginning new tablet copy session on tablet d7922fd756344d14a336ff0c54afdf73 from peer 6d34e50fb5dd4a1db58013d4863b8992 at {username='slave'} at 127.25.109.136:34777: session id = 6d34e50fb5dd4a1db58013d4863b8992-d7922fd756344d14a336ff0c54afdf73
I20240502 01:55:51.137874  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:51.140722  6224 tablet_copy_source_session.cc:205] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:51.148504  9491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7922fd756344d14a336ff0c54afdf73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:51.175729  6447 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:55:51.178329  8385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:51.185106  9598 tablet_bootstrap.cc:492] T 899963a8260a42c4b73fb80eda2ecaa9 P 9db17f84d37944098cbeb9e0d8456efb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:51.185648  9598 tablet_bootstrap.cc:492] T 899963a8260a42c4b73fb80eda2ecaa9 P 9db17f84d37944098cbeb9e0d8456efb: Bootstrap complete.
I20240502 01:55:51.185618  9491 tablet_copy_client.cc:783] T d7922fd756344d14a336ff0c54afdf73 P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:51.185927  9491 tablet_copy_client.cc:647] T d7922fd756344d14a336ff0c54afdf73 P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:51.185984  9598 ts_tablet_manager.cc:1367] T 899963a8260a42c4b73fb80eda2ecaa9 P 9db17f84d37944098cbeb9e0d8456efb: Time spent bootstrapping tablet: real 0.168s	user 0.042s	sys 0.009s
I20240502 01:55:51.187096  9598 raft_consensus.cc:348] T 899963a8260a42c4b73fb80eda2ecaa9 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:55:51.187393  9598 raft_consensus.cc:729] T 899963a8260a42c4b73fb80eda2ecaa9 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9db17f84d37944098cbeb9e0d8456efb, State: Initialized, Role: FOLLOWER
I20240502 01:55:51.192634  9491 tablet_copy_client.cc:523] T d7922fd756344d14a336ff0c54afdf73 P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:51.204993  9491 tablet_bootstrap.cc:492] T d7922fd756344d14a336ff0c54afdf73 P 6d34e50fb5dd4a1db58013d4863b8992: Bootstrap starting.
I20240502 01:55:51.223608  9598 consensus_queue.cc:260] T 899963a8260a42c4b73fb80eda2ecaa9 P 9db17f84d37944098cbeb9e0d8456efb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
W20240502 01:55:51.225009  6270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.129:36227, user_credentials={real_user=slave}} blocked reactor thread for 61241.8us
I20240502 01:55:51.228013  8001 raft_consensus.cc:1203] T ac3325a06d8841718158e1f5f181a398 P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:55:51.230687  6334 consensus_queue.cc:237] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 01:55:51.234784  9598 ts_tablet_manager.cc:1398] T 899963a8260a42c4b73fb80eda2ecaa9 P 9db17f84d37944098cbeb9e0d8456efb: Time spent starting tablet: real 0.049s	user 0.000s	sys 0.006s
I20240502 01:55:51.237455  6089 tablet_copy_service.cc:342] P 218bc8c1f9ce46059898fc738179adf9: Request end of tablet copy session 9db17f84d37944098cbeb9e0d8456efb-899963a8260a42c4b73fb80eda2ecaa9 received from {username='slave'} at 127.25.109.134:39709
I20240502 01:55:51.242959  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:51.259557  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:51.264397  6201 raft_consensus.cc:1261] T 3a7698f6bd4349ae832a22c853d76fea P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:55:51.260149  6270 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133:43379): Couldn't send request to peer 60298ffea1dd4c25a38765c2ec9ec9d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3a7698f6bd4349ae832a22c853d76fea. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:51.273190  9395 consensus_queue.cc:1035] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:51.279940  6089 tablet_copy_service.cc:433] P 218bc8c1f9ce46059898fc738179adf9: ending tablet copy session 9db17f84d37944098cbeb9e0d8456efb-899963a8260a42c4b73fb80eda2ecaa9 on tablet 899963a8260a42c4b73fb80eda2ecaa9 with peer 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:55:51.286617  6068 raft_consensus.cc:1261] T 3a7698f6bd4349ae832a22c853d76fea P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:51.287957  9705 consensus_queue.cc:1035] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:51.290761  6046 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:55:51.295002  6066 consensus_queue.cc:237] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:55:51.299803  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:51.313978  9705 raft_consensus.cc:2932] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:55:51.319597  6068 raft_consensus.cc:2932] T 3a7698f6bd4349ae832a22c853d76fea P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:55:51.322631  5916 catalog_manager.cc:5484] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 reported cstate change: config changed from index -1 to 4, VOTER 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133) added. New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:55:51.335197  6201 raft_consensus.cc:2932] T 3a7698f6bd4349ae832a22c853d76fea P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:55:51.335669  9636 tablet_bootstrap.cc:492] T 6984d638991442e3adaed1b24bee95e9 P 6d34e50fb5dd4a1db58013d4863b8992: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:51.341122  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:55:51.342979  9636 tablet_bootstrap.cc:492] T 6984d638991442e3adaed1b24bee95e9 P 6d34e50fb5dd4a1db58013d4863b8992: Bootstrap complete.
I20240502 01:55:51.344765  9636 ts_tablet_manager.cc:1367] T 6984d638991442e3adaed1b24bee95e9 P 6d34e50fb5dd4a1db58013d4863b8992: Time spent bootstrapping tablet: real 0.263s	user 0.045s	sys 0.015s
I20240502 01:55:51.347416  9636 raft_consensus.cc:348] T 6984d638991442e3adaed1b24bee95e9 P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:55:51.348194  9636 raft_consensus.cc:729] T 6984d638991442e3adaed1b24bee95e9 P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d34e50fb5dd4a1db58013d4863b8992, State: Initialized, Role: FOLLOWER
I20240502 01:55:51.348701  9636 consensus_queue.cc:260] T 6984d638991442e3adaed1b24bee95e9 P 6d34e50fb5dd4a1db58013d4863b8992 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:55:51.349764  9636 ts_tablet_manager.cc:1398] T 6984d638991442e3adaed1b24bee95e9 P 6d34e50fb5dd4a1db58013d4863b8992: Time spent starting tablet: real 0.005s	user 0.001s	sys 0.001s
I20240502 01:55:51.351156  6489 tablet_copy_service.cc:342] P 07cdb213c77c4ffda1f999b66b725399: Request end of tablet copy session 6d34e50fb5dd4a1db58013d4863b8992-6984d638991442e3adaed1b24bee95e9 received from {username='slave'} at 127.25.109.136:34855
I20240502 01:55:51.351310  6489 tablet_copy_service.cc:433] P 07cdb213c77c4ffda1f999b66b725399: ending tablet copy session 6d34e50fb5dd4a1db58013d4863b8992-6984d638991442e3adaed1b24bee95e9 on tablet 6984d638991442e3adaed1b24bee95e9 with peer 6d34e50fb5dd4a1db58013d4863b8992
I20240502 01:55:51.373350  9719 ts_tablet_manager.cc:902] T 99e74a49b42745888058db67049dc682 P 436822a397664c54b8d1b9eb3bc6cff4: Initiating tablet copy from peer 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227)
W20240502 01:55:51.406841  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c58d6b2401c047cd9af56a66c2e00105. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:51.407737  6467 raft_consensus.cc:1261] T c58d6b2401c047cd9af56a66c2e00105 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:51.408598  9535 consensus_queue.cc:1035] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:51.410261  6201 raft_consensus.cc:1261] T c58d6b2401c047cd9af56a66c2e00105 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:51.411028  8951 consensus_queue.cc:1035] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:51.449887  8951 raft_consensus.cc:2932] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:51.457113  6467 raft_consensus.cc:2932] T c58d6b2401c047cd9af56a66c2e00105 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:51.458521  6201 raft_consensus.cc:2932] T c58d6b2401c047cd9af56a66c2e00105 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:51.460290  8001 raft_consensus.cc:1203] T 4486b413e5c14ec6953e4895ee5797cf P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:55:51.466723  5921 catalog_manager.cc:5484] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 reported cstate change: config changed from index -1 to 4, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New cstate: current_term: 1 leader_uuid: "218bc8c1f9ce46059898fc738179adf9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:51.477344  6467 consensus_queue.cc:237] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
W20240502 01:55:51.509371  6401 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 73ed91df7ff04846991c95b30a86ea7e. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:51.510434  6066 raft_consensus.cc:1261] T 73ed91df7ff04846991c95b30a86ea7e P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:51.513603  6201 raft_consensus.cc:1261] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:51.513603  9217 consensus_queue.cc:1035] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:51.516868  9217 consensus_queue.cc:1035] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:51.537101  9217 raft_consensus.cc:2932] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:51.539733  9719 tablet_copy_client.cc:308] T 99e74a49b42745888058db67049dc682 P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.129:36227
I20240502 01:55:51.540683  6204 raft_consensus.cc:2932] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:51.551400  6066 raft_consensus.cc:2932] T 73ed91df7ff04846991c95b30a86ea7e P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:51.551815  5916 catalog_manager.cc:5484] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 reported cstate change: config changed from index -1 to 4, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New cstate: current_term: 1 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:51.582540  6089 tablet_copy_service.cc:140] P 218bc8c1f9ce46059898fc738179adf9: Received BeginTabletCopySession request for tablet 99e74a49b42745888058db67049dc682 from peer 436822a397664c54b8d1b9eb3bc6cff4 ({username='slave'} at 127.25.109.135:47139)
I20240502 01:55:51.582792  6089 tablet_copy_service.cc:161] P 218bc8c1f9ce46059898fc738179adf9: Beginning new tablet copy session on tablet 99e74a49b42745888058db67049dc682 from peer 436822a397664c54b8d1b9eb3bc6cff4 at {username='slave'} at 127.25.109.135:47139: session id = 436822a397664c54b8d1b9eb3bc6cff4-99e74a49b42745888058db67049dc682
I20240502 01:55:51.585695  6089 tablet_copy_source_session.cc:205] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:51.587437  9719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99e74a49b42745888058db67049dc682. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:51.598093  9719 tablet_copy_client.cc:783] T 99e74a49b42745888058db67049dc682 P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:51.598491  9719 tablet_copy_client.cc:647] T 99e74a49b42745888058db67049dc682 P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:51.600554  9719 tablet_copy_client.cc:523] T 99e74a49b42745888058db67049dc682 P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:51.614356  9719 tablet_bootstrap.cc:492] T 99e74a49b42745888058db67049dc682 P 436822a397664c54b8d1b9eb3bc6cff4: Bootstrap starting.
I20240502 01:55:51.648065  9491 tablet_bootstrap.cc:492] T d7922fd756344d14a336ff0c54afdf73 P 6d34e50fb5dd4a1db58013d4863b8992: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:51.648653  9491 tablet_bootstrap.cc:492] T d7922fd756344d14a336ff0c54afdf73 P 6d34e50fb5dd4a1db58013d4863b8992: Bootstrap complete.
I20240502 01:55:51.649016  9491 ts_tablet_manager.cc:1367] T d7922fd756344d14a336ff0c54afdf73 P 6d34e50fb5dd4a1db58013d4863b8992: Time spent bootstrapping tablet: real 0.444s	user 0.044s	sys 0.008s
I20240502 01:55:51.649891  9491 raft_consensus.cc:348] T d7922fd756344d14a336ff0c54afdf73 P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:55:51.650240  9491 raft_consensus.cc:729] T d7922fd756344d14a336ff0c54afdf73 P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d34e50fb5dd4a1db58013d4863b8992, State: Initialized, Role: FOLLOWER
I20240502 01:55:51.650590  9491 consensus_queue.cc:260] T d7922fd756344d14a336ff0c54afdf73 P 6d34e50fb5dd4a1db58013d4863b8992 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:55:51.651472  9491 ts_tablet_manager.cc:1398] T d7922fd756344d14a336ff0c54afdf73 P 6d34e50fb5dd4a1db58013d4863b8992: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:55:51.653154  6224 tablet_copy_service.cc:342] P a2c90ee440b44b708bccc2e2d5385cc0: Request end of tablet copy session 6d34e50fb5dd4a1db58013d4863b8992-d7922fd756344d14a336ff0c54afdf73 received from {username='slave'} at 127.25.109.136:34777
I20240502 01:55:51.653308  6224 tablet_copy_service.cc:433] P a2c90ee440b44b708bccc2e2d5385cc0: ending tablet copy session 6d34e50fb5dd4a1db58013d4863b8992-d7922fd756344d14a336ff0c54afdf73 on tablet d7922fd756344d14a336ff0c54afdf73 with peer 6d34e50fb5dd4a1db58013d4863b8992
I20240502 01:55:51.655761  6066 consensus_queue.cc:237] T a398da35f18c4df394db82c5bf89aeb1 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
W20240502 01:55:51.669596  5998 consensus_peers.cc:485] T a398da35f18c4df394db82c5bf89aeb1 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135:40543): Couldn't send request to peer 436822a397664c54b8d1b9eb3bc6cff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a398da35f18c4df394db82c5bf89aeb1. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:51.671053  6201 raft_consensus.cc:1261] T a398da35f18c4df394db82c5bf89aeb1 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:51.672102  8951 consensus_queue.cc:1035] T a398da35f18c4df394db82c5bf89aeb1 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:51.681739  6334 raft_consensus.cc:1261] T a398da35f18c4df394db82c5bf89aeb1 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:51.682539  8951 consensus_queue.cc:1035] T a398da35f18c4df394db82c5bf89aeb1 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:51.714956  9719 tablet_bootstrap.cc:492] T 99e74a49b42745888058db67049dc682 P 436822a397664c54b8d1b9eb3bc6cff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:51.715493  9719 tablet_bootstrap.cc:492] T 99e74a49b42745888058db67049dc682 P 436822a397664c54b8d1b9eb3bc6cff4: Bootstrap complete.
I20240502 01:55:51.715801  9719 ts_tablet_manager.cc:1367] T 99e74a49b42745888058db67049dc682 P 436822a397664c54b8d1b9eb3bc6cff4: Time spent bootstrapping tablet: real 0.105s	user 0.041s	sys 0.007s
I20240502 01:55:51.716730  9719 raft_consensus.cc:348] T 99e74a49b42745888058db67049dc682 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:55:51.717017  9719 raft_consensus.cc:729] T 99e74a49b42745888058db67049dc682 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 436822a397664c54b8d1b9eb3bc6cff4, State: Initialized, Role: FOLLOWER
I20240502 01:55:51.717209  8135 raft_consensus.cc:1203] T 899963a8260a42c4b73fb80eda2ecaa9 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:55:51.717273  9719 consensus_queue.cc:260] T 99e74a49b42745888058db67049dc682 P 436822a397664c54b8d1b9eb3bc6cff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:55:51.724578  9719 ts_tablet_manager.cc:1398] T 99e74a49b42745888058db67049dc682 P 436822a397664c54b8d1b9eb3bc6cff4: Time spent starting tablet: real 0.009s	user 0.001s	sys 0.003s
I20240502 01:55:51.726073  6089 tablet_copy_service.cc:342] P 218bc8c1f9ce46059898fc738179adf9: Request end of tablet copy session 436822a397664c54b8d1b9eb3bc6cff4-99e74a49b42745888058db67049dc682 received from {username='slave'} at 127.25.109.135:47139
I20240502 01:55:51.726227  6089 tablet_copy_service.cc:433] P 218bc8c1f9ce46059898fc738179adf9: ending tablet copy session 436822a397664c54b8d1b9eb3bc6cff4-99e74a49b42745888058db67049dc682 on tablet 99e74a49b42745888058db67049dc682 with peer 436822a397664c54b8d1b9eb3bc6cff4
I20240502 01:55:51.730154  8951 raft_consensus.cc:2932] T a398da35f18c4df394db82c5bf89aeb1 P 218bc8c1f9ce46059898fc738179adf9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:55:51.735070  6201 raft_consensus.cc:2932] T a398da35f18c4df394db82c5bf89aeb1 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:55:51.745349  6334 raft_consensus.cc:2932] T a398da35f18c4df394db82c5bf89aeb1 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:55:51.756222  9799 ts_tablet_manager.cc:902] T 3a7698f6bd4349ae832a22c853d76fea P 60298ffea1dd4c25a38765c2ec9ec9d5: Initiating tablet copy from peer 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245)
I20240502 01:55:51.756520  5913 catalog_manager.cc:5484] T a398da35f18c4df394db82c5bf89aeb1 P a2c90ee440b44b708bccc2e2d5385cc0 reported cstate change: config changed from index -1 to 4, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New cstate: current_term: 1 leader_uuid: "218bc8c1f9ce46059898fc738179adf9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:55:51.782634  9799 tablet_copy_client.cc:308] T 3a7698f6bd4349ae832a22c853d76fea P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.131:44245
I20240502 01:55:51.783979  6357 tablet_copy_service.cc:140] P 94ffd7aaf85b4e17881241b8396fa2f3: Received BeginTabletCopySession request for tablet 3a7698f6bd4349ae832a22c853d76fea from peer 60298ffea1dd4c25a38765c2ec9ec9d5 ({username='slave'} at 127.25.109.133:43975)
I20240502 01:55:51.784197  6357 tablet_copy_service.cc:161] P 94ffd7aaf85b4e17881241b8396fa2f3: Beginning new tablet copy session on tablet 3a7698f6bd4349ae832a22c853d76fea from peer 60298ffea1dd4c25a38765c2ec9ec9d5 at {username='slave'} at 127.25.109.133:43975: session id = 60298ffea1dd4c25a38765c2ec9ec9d5-3a7698f6bd4349ae832a22c853d76fea
I20240502 01:55:51.787000  6357 tablet_copy_source_session.cc:205] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:51.789791  9799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a7698f6bd4349ae832a22c853d76fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:51.796538  9799 tablet_copy_client.cc:783] T 3a7698f6bd4349ae832a22c853d76fea P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:51.814687  9799 tablet_copy_client.cc:647] T 3a7698f6bd4349ae832a22c853d76fea P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:51.818113  9799 tablet_copy_client.cc:523] T 3a7698f6bd4349ae832a22c853d76fea P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:51.828089  8406 raft_consensus.cc:1203] T 6984d638991442e3adaed1b24bee95e9 P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:55:51.829025  9799 tablet_bootstrap.cc:492] T 3a7698f6bd4349ae832a22c853d76fea P 60298ffea1dd4c25a38765c2ec9ec9d5: Bootstrap starting.
I20240502 01:55:51.831142  6467 consensus_queue.cc:237] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:55:51.832023  6447 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:55:51.841688  8270 raft_consensus.cc:1203] T 99e74a49b42745888058db67049dc682 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:55:51.843585  6334 raft_consensus.cc:1261] T 4eb91f5df851459b8ecf09cbe34fefa5 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:55:51.844756  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eb91f5df851459b8ecf09cbe34fefa5. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:51.845158  6046 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:55:51.845988  9317 consensus_queue.cc:1035] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:51.870272  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:51.872337  6201 raft_consensus.cc:1261] T 4eb91f5df851459b8ecf09cbe34fefa5 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:51.882192  9317 consensus_queue.cc:1035] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:51.966893  9217 raft_consensus.cc:2932] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:51.976519  6204 raft_consensus.cc:2932] T 4eb91f5df851459b8ecf09cbe34fefa5 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:51.992594  5916 catalog_manager.cc:5484] T 4eb91f5df851459b8ecf09cbe34fefa5 P a2c90ee440b44b708bccc2e2d5385cc0 reported cstate change: config changed from index -1 to 4, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New cstate: current_term: 1 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:52.008570  8385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:52.012893  6334 raft_consensus.cc:2932] T 4eb91f5df851459b8ecf09cbe34fefa5 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:52.032542  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:55:52.037865  9491 ts_tablet_manager.cc:902] T 73ed91df7ff04846991c95b30a86ea7e P 6d34e50fb5dd4a1db58013d4863b8992: Initiating tablet copy from peer 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867)
I20240502 01:55:52.064973  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:52.069023  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:52.091512  9860 ts_tablet_manager.cc:902] T c58d6b2401c047cd9af56a66c2e00105 P 9db17f84d37944098cbeb9e0d8456efb: Initiating tablet copy from peer 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227)
I20240502 01:55:52.096590  9491 tablet_copy_client.cc:308] T 73ed91df7ff04846991c95b30a86ea7e P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.132:36867
I20240502 01:55:52.097522  6489 tablet_copy_service.cc:140] P 07cdb213c77c4ffda1f999b66b725399: Received BeginTabletCopySession request for tablet 73ed91df7ff04846991c95b30a86ea7e from peer 6d34e50fb5dd4a1db58013d4863b8992 ({username='slave'} at 127.25.109.136:34855)
I20240502 01:55:52.097712  6489 tablet_copy_service.cc:161] P 07cdb213c77c4ffda1f999b66b725399: Beginning new tablet copy session on tablet 73ed91df7ff04846991c95b30a86ea7e from peer 6d34e50fb5dd4a1db58013d4863b8992 at {username='slave'} at 127.25.109.136:34855: session id = 6d34e50fb5dd4a1db58013d4863b8992-73ed91df7ff04846991c95b30a86ea7e
I20240502 01:55:52.101617  6489 tablet_copy_source_session.cc:205] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:52.104470  9491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73ed91df7ff04846991c95b30a86ea7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:52.121661  8406 raft_consensus.cc:1203] T d7922fd756344d14a336ff0c54afdf73 P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:55:52.136219  9719 ts_tablet_manager.cc:902] T a398da35f18c4df394db82c5bf89aeb1 P 436822a397664c54b8d1b9eb3bc6cff4: Initiating tablet copy from peer 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227)
I20240502 01:55:52.140516  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:52.157738  9491 tablet_copy_client.cc:783] T 73ed91df7ff04846991c95b30a86ea7e P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:52.158151  9491 tablet_copy_client.cc:647] T 73ed91df7ff04846991c95b30a86ea7e P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:52.167928  9491 tablet_copy_client.cc:523] T 73ed91df7ff04846991c95b30a86ea7e P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:52.173709  9860 tablet_copy_client.cc:308] T c58d6b2401c047cd9af56a66c2e00105 P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.129:36227
I20240502 01:55:52.174749  6089 tablet_copy_service.cc:140] P 218bc8c1f9ce46059898fc738179adf9: Received BeginTabletCopySession request for tablet c58d6b2401c047cd9af56a66c2e00105 from peer 9db17f84d37944098cbeb9e0d8456efb ({username='slave'} at 127.25.109.134:39709)
I20240502 01:55:52.174980  6089 tablet_copy_service.cc:161] P 218bc8c1f9ce46059898fc738179adf9: Beginning new tablet copy session on tablet c58d6b2401c047cd9af56a66c2e00105 from peer 9db17f84d37944098cbeb9e0d8456efb at {username='slave'} at 127.25.109.134:39709: session id = 9db17f84d37944098cbeb9e0d8456efb-c58d6b2401c047cd9af56a66c2e00105
I20240502 01:55:52.178648  6089 tablet_copy_source_session.cc:205] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:52.180842  9860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c58d6b2401c047cd9af56a66c2e00105. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:52.229228  9491 tablet_bootstrap.cc:492] T 73ed91df7ff04846991c95b30a86ea7e P 6d34e50fb5dd4a1db58013d4863b8992: Bootstrap starting.
I20240502 01:55:52.257751  9860 tablet_copy_client.cc:783] T c58d6b2401c047cd9af56a66c2e00105 P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:52.258251  9719 tablet_copy_client.cc:308] T a398da35f18c4df394db82c5bf89aeb1 P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.129:36227
I20240502 01:55:52.259024  6089 tablet_copy_service.cc:140] P 218bc8c1f9ce46059898fc738179adf9: Received BeginTabletCopySession request for tablet a398da35f18c4df394db82c5bf89aeb1 from peer 436822a397664c54b8d1b9eb3bc6cff4 ({username='slave'} at 127.25.109.135:47139)
I20240502 01:55:52.259244  6089 tablet_copy_service.cc:161] P 218bc8c1f9ce46059898fc738179adf9: Beginning new tablet copy session on tablet a398da35f18c4df394db82c5bf89aeb1 from peer 436822a397664c54b8d1b9eb3bc6cff4 at {username='slave'} at 127.25.109.135:47139: session id = 436822a397664c54b8d1b9eb3bc6cff4-a398da35f18c4df394db82c5bf89aeb1
I20240502 01:55:52.262243  6089 tablet_copy_source_session.cc:205] T a398da35f18c4df394db82c5bf89aeb1 P 218bc8c1f9ce46059898fc738179adf9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:52.265166  9719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a398da35f18c4df394db82c5bf89aeb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:52.271626  6467 consensus_queue.cc:237] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:55:52.275607  6204 raft_consensus.cc:1261] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 01:55:52.276862  6401 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:52.276897  9719 tablet_copy_client.cc:783] T a398da35f18c4df394db82c5bf89aeb1 P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:52.277355  9719 tablet_copy_client.cc:647] T a398da35f18c4df394db82c5bf89aeb1 P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:52.277379  9317 consensus_queue.cc:1035] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:52.279732  9719 tablet_copy_client.cc:523] T a398da35f18c4df394db82c5bf89aeb1 P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:52.279968  6066 raft_consensus.cc:1261] T 73ed91df7ff04846991c95b30a86ea7e P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:52.280993  9217 consensus_queue.cc:1035] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:52.283792  9719 tablet_bootstrap.cc:492] T a398da35f18c4df394db82c5bf89aeb1 P 436822a397664c54b8d1b9eb3bc6cff4: Bootstrap starting.
I20240502 01:55:52.289327  9317 raft_consensus.cc:2932] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:55:52.290884  9860 tablet_copy_client.cc:647] T c58d6b2401c047cd9af56a66c2e00105 P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:52.293089  9860 tablet_copy_client.cc:523] T c58d6b2401c047cd9af56a66c2e00105 P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:52.293628  6066 raft_consensus.cc:2932] T 73ed91df7ff04846991c95b30a86ea7e P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:55:52.296890  9860 tablet_bootstrap.cc:492] T c58d6b2401c047cd9af56a66c2e00105 P 9db17f84d37944098cbeb9e0d8456efb: Bootstrap starting.
I20240502 01:55:52.309513  5913 catalog_manager.cc:5484] T 73ed91df7ff04846991c95b30a86ea7e P 218bc8c1f9ce46059898fc738179adf9 reported cstate change: config changed from index 4 to 5, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New cstate: current_term: 1 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:55:52.332628  6201 raft_consensus.cc:2932] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:55:52.362321  9719 tablet_bootstrap.cc:492] T a398da35f18c4df394db82c5bf89aeb1 P 436822a397664c54b8d1b9eb3bc6cff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:52.362931  9719 tablet_bootstrap.cc:492] T a398da35f18c4df394db82c5bf89aeb1 P 436822a397664c54b8d1b9eb3bc6cff4: Bootstrap complete.
I20240502 01:55:52.363301  9719 ts_tablet_manager.cc:1367] T a398da35f18c4df394db82c5bf89aeb1 P 436822a397664c54b8d1b9eb3bc6cff4: Time spent bootstrapping tablet: real 0.080s	user 0.043s	sys 0.016s
I20240502 01:55:52.364337  9719 raft_consensus.cc:348] T a398da35f18c4df394db82c5bf89aeb1 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:55:52.364672  9719 raft_consensus.cc:729] T a398da35f18c4df394db82c5bf89aeb1 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 436822a397664c54b8d1b9eb3bc6cff4, State: Initialized, Role: FOLLOWER
I20240502 01:55:52.365171  9719 consensus_queue.cc:260] T a398da35f18c4df394db82c5bf89aeb1 P 436822a397664c54b8d1b9eb3bc6cff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
W20240502 01:55:52.367435  6400 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135:40543): Couldn't send request to peer 436822a397664c54b8d1b9eb3bc6cff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 73ed91df7ff04846991c95b30a86ea7e. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:52.375272  9903 ts_tablet_manager.cc:902] T 4eb91f5df851459b8ecf09cbe34fefa5 P 9db17f84d37944098cbeb9e0d8456efb: Initiating tablet copy from peer 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867)
I20240502 01:55:52.376281  9719 ts_tablet_manager.cc:1398] T a398da35f18c4df394db82c5bf89aeb1 P 436822a397664c54b8d1b9eb3bc6cff4: Time spent starting tablet: real 0.013s	user 0.004s	sys 0.004s
I20240502 01:55:52.377748  6089 tablet_copy_service.cc:342] P 218bc8c1f9ce46059898fc738179adf9: Request end of tablet copy session 436822a397664c54b8d1b9eb3bc6cff4-a398da35f18c4df394db82c5bf89aeb1 received from {username='slave'} at 127.25.109.135:47139
I20240502 01:55:52.377905  6089 tablet_copy_service.cc:433] P 218bc8c1f9ce46059898fc738179adf9: ending tablet copy session 436822a397664c54b8d1b9eb3bc6cff4-a398da35f18c4df394db82c5bf89aeb1 on tablet a398da35f18c4df394db82c5bf89aeb1 with peer 436822a397664c54b8d1b9eb3bc6cff4
I20240502 01:55:52.391139  9903 tablet_copy_client.cc:308] T 4eb91f5df851459b8ecf09cbe34fefa5 P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.132:36867
I20240502 01:55:52.392113  6489 tablet_copy_service.cc:140] P 07cdb213c77c4ffda1f999b66b725399: Received BeginTabletCopySession request for tablet 4eb91f5df851459b8ecf09cbe34fefa5 from peer 9db17f84d37944098cbeb9e0d8456efb ({username='slave'} at 127.25.109.134:56199)
I20240502 01:55:52.392321  6489 tablet_copy_service.cc:161] P 07cdb213c77c4ffda1f999b66b725399: Beginning new tablet copy session on tablet 4eb91f5df851459b8ecf09cbe34fefa5 from peer 9db17f84d37944098cbeb9e0d8456efb at {username='slave'} at 127.25.109.134:56199: session id = 9db17f84d37944098cbeb9e0d8456efb-4eb91f5df851459b8ecf09cbe34fefa5
I20240502 01:55:52.395237  6489 tablet_copy_source_session.cc:205] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:52.410012  6201 consensus_queue.cc:237] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:55:52.410604  9903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4eb91f5df851459b8ecf09cbe34fefa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:52.426687  9903 tablet_copy_client.cc:783] T 4eb91f5df851459b8ecf09cbe34fefa5 P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:52.427035  9903 tablet_copy_client.cc:647] T 4eb91f5df851459b8ecf09cbe34fefa5 P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:52.440371  9799 tablet_bootstrap.cc:492] T 3a7698f6bd4349ae832a22c853d76fea P 60298ffea1dd4c25a38765c2ec9ec9d5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:52.440636  9903 tablet_copy_client.cc:523] T 4eb91f5df851459b8ecf09cbe34fefa5 P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:52.447330  9903 tablet_bootstrap.cc:492] T 4eb91f5df851459b8ecf09cbe34fefa5 P 9db17f84d37944098cbeb9e0d8456efb: Bootstrap starting.
I20240502 01:55:52.461324  9799 tablet_bootstrap.cc:492] T 3a7698f6bd4349ae832a22c853d76fea P 60298ffea1dd4c25a38765c2ec9ec9d5: Bootstrap complete.
I20240502 01:55:52.464455  9799 ts_tablet_manager.cc:1367] T 3a7698f6bd4349ae832a22c853d76fea P 60298ffea1dd4c25a38765c2ec9ec9d5: Time spent bootstrapping tablet: real 0.636s	user 0.044s	sys 0.010s
I20240502 01:55:52.468575  6467 raft_consensus.cc:1261] T ca458907378e42d48adaa0ee647db4cd P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Refusing update from remote peer a2c90ee440b44b708bccc2e2d5385cc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:52.469506  9616 consensus_queue.cc:1035] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:52.470818  9799 raft_consensus.cc:348] T 3a7698f6bd4349ae832a22c853d76fea P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 01:55:52.472419  9799 raft_consensus.cc:729] T 3a7698f6bd4349ae832a22c853d76fea P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60298ffea1dd4c25a38765c2ec9ec9d5, State: Initialized, Role: FOLLOWER
I20240502 01:55:52.474061  9799 consensus_queue.cc:260] T 3a7698f6bd4349ae832a22c853d76fea P 60298ffea1dd4c25a38765c2ec9ec9d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 01:55:52.486892  6066 raft_consensus.cc:1261] T ca458907378e42d48adaa0ee647db4cd P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Refusing update from remote peer a2c90ee440b44b708bccc2e2d5385cc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:52.508148  9616 consensus_queue.cc:1035] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:55:52.520165  9926 log.cc:924] Time spent T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20240502 01:55:52.540319  9799 ts_tablet_manager.cc:1398] T 3a7698f6bd4349ae832a22c853d76fea P 60298ffea1dd4c25a38765c2ec9ec9d5: Time spent starting tablet: real 0.076s	user 0.006s	sys 0.006s
I20240502 01:55:52.544911  6357 tablet_copy_service.cc:342] P 94ffd7aaf85b4e17881241b8396fa2f3: Request end of tablet copy session 60298ffea1dd4c25a38765c2ec9ec9d5-3a7698f6bd4349ae832a22c853d76fea received from {username='slave'} at 127.25.109.133:43975
I20240502 01:55:52.545066  6357 tablet_copy_service.cc:433] P 94ffd7aaf85b4e17881241b8396fa2f3: ending tablet copy session 60298ffea1dd4c25a38765c2ec9ec9d5-3a7698f6bd4349ae832a22c853d76fea on tablet 3a7698f6bd4349ae832a22c853d76fea with peer 60298ffea1dd4c25a38765c2ec9ec9d5
I20240502 01:55:52.548266  8270 raft_consensus.cc:1203] T a398da35f18c4df394db82c5bf89aeb1 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
W20240502 01:55:52.582756  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ca458907378e42d48adaa0ee647db4cd. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:52.594847  9945 raft_consensus.cc:2932] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:52.602622  6066 raft_consensus.cc:2932] T ca458907378e42d48adaa0ee647db4cd P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:52.606815  9860 tablet_bootstrap.cc:492] T c58d6b2401c047cd9af56a66c2e00105 P 9db17f84d37944098cbeb9e0d8456efb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:52.609197  6467 raft_consensus.cc:2932] T ca458907378e42d48adaa0ee647db4cd P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:52.611146  5916 catalog_manager.cc:5484] T ca458907378e42d48adaa0ee647db4cd P 218bc8c1f9ce46059898fc738179adf9 reported cstate change: config changed from index -1 to 4, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New cstate: current_term: 1 leader_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:52.616003  9860 tablet_bootstrap.cc:492] T c58d6b2401c047cd9af56a66c2e00105 P 9db17f84d37944098cbeb9e0d8456efb: Bootstrap complete.
I20240502 01:55:52.616539  9860 ts_tablet_manager.cc:1367] T c58d6b2401c047cd9af56a66c2e00105 P 9db17f84d37944098cbeb9e0d8456efb: Time spent bootstrapping tablet: real 0.320s	user 0.046s	sys 0.004s
I20240502 01:55:52.617722  9860 raft_consensus.cc:348] T c58d6b2401c047cd9af56a66c2e00105 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:55:52.618031  9860 raft_consensus.cc:729] T c58d6b2401c047cd9af56a66c2e00105 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9db17f84d37944098cbeb9e0d8456efb, State: Initialized, Role: FOLLOWER
I20240502 01:55:52.618394  9860 consensus_queue.cc:260] T c58d6b2401c047cd9af56a66c2e00105 P 9db17f84d37944098cbeb9e0d8456efb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:55:52.622454  6201 consensus_queue.cc:237] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:55:52.628196  9860 ts_tablet_manager.cc:1398] T c58d6b2401c047cd9af56a66c2e00105 P 9db17f84d37944098cbeb9e0d8456efb: Time spent starting tablet: real 0.011s	user 0.004s	sys 0.004s
I20240502 01:55:52.630997  6089 tablet_copy_service.cc:342] P 218bc8c1f9ce46059898fc738179adf9: Request end of tablet copy session 9db17f84d37944098cbeb9e0d8456efb-c58d6b2401c047cd9af56a66c2e00105 received from {username='slave'} at 127.25.109.134:39709
I20240502 01:55:52.631160  6089 tablet_copy_service.cc:433] P 218bc8c1f9ce46059898fc738179adf9: ending tablet copy session 9db17f84d37944098cbeb9e0d8456efb-c58d6b2401c047cd9af56a66c2e00105 on tablet c58d6b2401c047cd9af56a66c2e00105 with peer 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:55:52.632350  9491 tablet_bootstrap.cc:492] T 73ed91df7ff04846991c95b30a86ea7e P 6d34e50fb5dd4a1db58013d4863b8992: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:52.637800  9491 tablet_bootstrap.cc:492] T 73ed91df7ff04846991c95b30a86ea7e P 6d34e50fb5dd4a1db58013d4863b8992: Bootstrap complete.
I20240502 01:55:52.638128  9491 ts_tablet_manager.cc:1367] T 73ed91df7ff04846991c95b30a86ea7e P 6d34e50fb5dd4a1db58013d4863b8992: Time spent bootstrapping tablet: real 0.409s	user 0.053s	sys 0.003s
I20240502 01:55:52.639006  9491 raft_consensus.cc:348] T 73ed91df7ff04846991c95b30a86ea7e P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:55:52.647806  9491 raft_consensus.cc:729] T 73ed91df7ff04846991c95b30a86ea7e P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d34e50fb5dd4a1db58013d4863b8992, State: Initialized, Role: FOLLOWER
I20240502 01:55:52.648229  9491 consensus_queue.cc:260] T 73ed91df7ff04846991c95b30a86ea7e P 6d34e50fb5dd4a1db58013d4863b8992 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
W20240502 01:55:52.687005  6134 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135:40543): Couldn't send request to peer 436822a397664c54b8d1b9eb3bc6cff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 086509f1ee0a4a81b2f8194b50dbe159. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:52.688446  6467 raft_consensus.cc:1261] T 086509f1ee0a4a81b2f8194b50dbe159 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Refusing update from remote peer a2c90ee440b44b708bccc2e2d5385cc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:52.689231  9945 consensus_queue.cc:1035] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:52.698632  6066 raft_consensus.cc:1261] T 086509f1ee0a4a81b2f8194b50dbe159 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Refusing update from remote peer a2c90ee440b44b708bccc2e2d5385cc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:52.700101  9945 consensus_queue.cc:1035] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:52.702893  9903 tablet_bootstrap.cc:492] T 4eb91f5df851459b8ecf09cbe34fefa5 P 9db17f84d37944098cbeb9e0d8456efb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:52.703405  9903 tablet_bootstrap.cc:492] T 4eb91f5df851459b8ecf09cbe34fefa5 P 9db17f84d37944098cbeb9e0d8456efb: Bootstrap complete.
I20240502 01:55:52.703712  9903 ts_tablet_manager.cc:1367] T 4eb91f5df851459b8ecf09cbe34fefa5 P 9db17f84d37944098cbeb9e0d8456efb: Time spent bootstrapping tablet: real 0.256s	user 0.046s	sys 0.013s
I20240502 01:55:52.704613  9903 raft_consensus.cc:348] T 4eb91f5df851459b8ecf09cbe34fefa5 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:55:52.704875  9903 raft_consensus.cc:729] T 4eb91f5df851459b8ecf09cbe34fefa5 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9db17f84d37944098cbeb9e0d8456efb, State: Initialized, Role: FOLLOWER
I20240502 01:55:52.708806  9903 consensus_queue.cc:260] T 4eb91f5df851459b8ecf09cbe34fefa5 P 9db17f84d37944098cbeb9e0d8456efb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:55:52.709511  9903 ts_tablet_manager.cc:1398] T 4eb91f5df851459b8ecf09cbe34fefa5 P 9db17f84d37944098cbeb9e0d8456efb: Time spent starting tablet: real 0.006s	user 0.000s	sys 0.004s
I20240502 01:55:52.714607  6489 tablet_copy_service.cc:342] P 07cdb213c77c4ffda1f999b66b725399: Request end of tablet copy session 9db17f84d37944098cbeb9e0d8456efb-4eb91f5df851459b8ecf09cbe34fefa5 received from {username='slave'} at 127.25.109.134:56199
I20240502 01:55:52.714763  6489 tablet_copy_service.cc:433] P 07cdb213c77c4ffda1f999b66b725399: ending tablet copy session 9db17f84d37944098cbeb9e0d8456efb-4eb91f5df851459b8ecf09cbe34fefa5 on tablet 4eb91f5df851459b8ecf09cbe34fefa5 with peer 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:55:52.730329  9945 raft_consensus.cc:2932] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:55:52.740128  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:52.741199  6467 raft_consensus.cc:2932] T 086509f1ee0a4a81b2f8194b50dbe159 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:55:52.741508  6066 raft_consensus.cc:2932] T 086509f1ee0a4a81b2f8194b50dbe159 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:55:52.743516  9491 ts_tablet_manager.cc:1398] T 73ed91df7ff04846991c95b30a86ea7e P 6d34e50fb5dd4a1db58013d4863b8992: Time spent starting tablet: real 0.105s	user 0.023s	sys 0.022s
I20240502 01:55:52.747494  6489 tablet_copy_service.cc:342] P 07cdb213c77c4ffda1f999b66b725399: Request end of tablet copy session 6d34e50fb5dd4a1db58013d4863b8992-73ed91df7ff04846991c95b30a86ea7e received from {username='slave'} at 127.25.109.136:34855
I20240502 01:55:52.747682  6489 tablet_copy_service.cc:433] P 07cdb213c77c4ffda1f999b66b725399: ending tablet copy session 6d34e50fb5dd4a1db58013d4863b8992-73ed91df7ff04846991c95b30a86ea7e on tablet 73ed91df7ff04846991c95b30a86ea7e with peer 6d34e50fb5dd4a1db58013d4863b8992
I20240502 01:55:52.750216  5920 catalog_manager.cc:5484] T 086509f1ee0a4a81b2f8194b50dbe159 P 218bc8c1f9ce46059898fc738179adf9 reported cstate change: config changed from index -1 to 4, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New cstate: current_term: 1 leader_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:55:52.764225  6447 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:55:52.765935  8406 raft_consensus.cc:2932] T 73ed91df7ff04846991c95b30a86ea7e P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:55:52.786038  6334 consensus_queue.cc:237] T 74eadf4c939c4c6383732cd74c5ec9d6 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 01:55:52.809886  6066 raft_consensus.cc:1261] T 74eadf4c939c4c6383732cd74c5ec9d6 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:55:52.810472  6270 consensus_peers.cc:485] T 74eadf4c939c4c6383732cd74c5ec9d6 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133:43379): Couldn't send request to peer 60298ffea1dd4c25a38765c2ec9ec9d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 74eadf4c939c4c6383732cd74c5ec9d6. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:52.810998  9705 consensus_queue.cc:1035] T 74eadf4c939c4c6383732cd74c5ec9d6 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:52.814122  6201 raft_consensus.cc:1261] T 74eadf4c939c4c6383732cd74c5ec9d6 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:52.814813  9705 consensus_queue.cc:1035] T 74eadf4c939c4c6383732cd74c5ec9d6 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:52.820019  8135 raft_consensus.cc:1203] T 4eb91f5df851459b8ecf09cbe34fefa5 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:55:52.819842  6046 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:55:52.831581  8001 raft_consensus.cc:1203] T 3a7698f6bd4349ae832a22c853d76fea P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:55:52.840035  9705 raft_consensus.cc:2932] T 74eadf4c939c4c6383732cd74c5ec9d6 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:55:52.841871  6204 raft_consensus.cc:2932] T 74eadf4c939c4c6383732cd74c5ec9d6 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:55:52.845583  6066 raft_consensus.cc:2932] T 74eadf4c939c4c6383732cd74c5ec9d6 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:55:52.849400  5916 catalog_manager.cc:5484] T 74eadf4c939c4c6383732cd74c5ec9d6 P a2c90ee440b44b708bccc2e2d5385cc0 reported cstate change: config changed from index -1 to 4, VOTER 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133) added. New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:55:52.852517  8135 raft_consensus.cc:1203] T c58d6b2401c047cd9af56a66c2e00105 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:55:52.889809  8385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:52.889981  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:52.912560 10003 ts_tablet_manager.cc:902] T 73ed91df7ff04846991c95b30a86ea7e P 436822a397664c54b8d1b9eb3bc6cff4: Initiating tablet copy from peer 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867)
I20240502 01:55:52.951148  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:52.953404  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:55:52.954631  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:53.081588  9903 ts_tablet_manager.cc:902] T ca458907378e42d48adaa0ee647db4cd P 9db17f84d37944098cbeb9e0d8456efb: Initiating tablet copy from peer a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197)
I20240502 01:55:53.112233 10003 tablet_copy_client.cc:308] T 73ed91df7ff04846991c95b30a86ea7e P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.132:36867
I20240502 01:55:53.135586  9903 tablet_copy_client.cc:308] T ca458907378e42d48adaa0ee647db4cd P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.130:38197
I20240502 01:55:53.149322  6224 tablet_copy_service.cc:140] P a2c90ee440b44b708bccc2e2d5385cc0: Received BeginTabletCopySession request for tablet ca458907378e42d48adaa0ee647db4cd from peer 9db17f84d37944098cbeb9e0d8456efb ({username='slave'} at 127.25.109.134:39927)
I20240502 01:55:53.149554  6224 tablet_copy_service.cc:161] P a2c90ee440b44b708bccc2e2d5385cc0: Beginning new tablet copy session on tablet ca458907378e42d48adaa0ee647db4cd from peer 9db17f84d37944098cbeb9e0d8456efb at {username='slave'} at 127.25.109.134:39927: session id = 9db17f84d37944098cbeb9e0d8456efb-ca458907378e42d48adaa0ee647db4cd
I20240502 01:55:53.153414  6224 tablet_copy_source_session.cc:205] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:53.155172  9903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca458907378e42d48adaa0ee647db4cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:53.162668  9903 tablet_copy_client.cc:783] T ca458907378e42d48adaa0ee647db4cd P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:53.162966  9903 tablet_copy_client.cc:647] T ca458907378e42d48adaa0ee647db4cd P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:53.165311  9903 tablet_copy_client.cc:523] T ca458907378e42d48adaa0ee647db4cd P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:53.170588  9903 tablet_bootstrap.cc:492] T ca458907378e42d48adaa0ee647db4cd P 9db17f84d37944098cbeb9e0d8456efb: Bootstrap starting.
I20240502 01:55:53.195724 10044 ts_tablet_manager.cc:902] T 086509f1ee0a4a81b2f8194b50dbe159 P 436822a397664c54b8d1b9eb3bc6cff4: Initiating tablet copy from peer a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197)
I20240502 01:55:53.204699  6489 tablet_copy_service.cc:140] P 07cdb213c77c4ffda1f999b66b725399: Received BeginTabletCopySession request for tablet 73ed91df7ff04846991c95b30a86ea7e from peer 436822a397664c54b8d1b9eb3bc6cff4 ({username='slave'} at 127.25.109.135:41247)
I20240502 01:55:53.204900  6489 tablet_copy_service.cc:161] P 07cdb213c77c4ffda1f999b66b725399: Beginning new tablet copy session on tablet 73ed91df7ff04846991c95b30a86ea7e from peer 436822a397664c54b8d1b9eb3bc6cff4 at {username='slave'} at 127.25.109.135:41247: session id = 436822a397664c54b8d1b9eb3bc6cff4-73ed91df7ff04846991c95b30a86ea7e
I20240502 01:55:53.215618  6489 tablet_copy_source_session.cc:205] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:53.217569 10003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73ed91df7ff04846991c95b30a86ea7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:53.224545 10003 tablet_copy_client.cc:783] T 73ed91df7ff04846991c95b30a86ea7e P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:53.228178 10003 tablet_copy_client.cc:647] T 73ed91df7ff04846991c95b30a86ea7e P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:53.230423 10003 tablet_copy_client.cc:523] T 73ed91df7ff04846991c95b30a86ea7e P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:53.234753 10003 tablet_bootstrap.cc:492] T 73ed91df7ff04846991c95b30a86ea7e P 436822a397664c54b8d1b9eb3bc6cff4: Bootstrap starting.
I20240502 01:55:53.267817  6334 consensus_queue.cc:237] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:55:53.273355  9903 tablet_bootstrap.cc:492] T ca458907378e42d48adaa0ee647db4cd P 9db17f84d37944098cbeb9e0d8456efb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:53.273913  9903 tablet_bootstrap.cc:492] T ca458907378e42d48adaa0ee647db4cd P 9db17f84d37944098cbeb9e0d8456efb: Bootstrap complete.
I20240502 01:55:53.274255  9903 ts_tablet_manager.cc:1367] T ca458907378e42d48adaa0ee647db4cd P 9db17f84d37944098cbeb9e0d8456efb: Time spent bootstrapping tablet: real 0.104s	user 0.042s	sys 0.009s
I20240502 01:55:53.275174  9903 raft_consensus.cc:348] T ca458907378e42d48adaa0ee647db4cd P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:55:53.275455  9903 raft_consensus.cc:729] T ca458907378e42d48adaa0ee647db4cd P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9db17f84d37944098cbeb9e0d8456efb, State: Initialized, Role: FOLLOWER
I20240502 01:55:53.275794  9903 consensus_queue.cc:260] T ca458907378e42d48adaa0ee647db4cd P 9db17f84d37944098cbeb9e0d8456efb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:55:53.279834  9903 ts_tablet_manager.cc:1398] T ca458907378e42d48adaa0ee647db4cd P 9db17f84d37944098cbeb9e0d8456efb: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20240502 01:55:53.283524  6224 tablet_copy_service.cc:342] P a2c90ee440b44b708bccc2e2d5385cc0: Request end of tablet copy session 9db17f84d37944098cbeb9e0d8456efb-ca458907378e42d48adaa0ee647db4cd received from {username='slave'} at 127.25.109.134:39927
I20240502 01:55:53.283737  6224 tablet_copy_service.cc:433] P a2c90ee440b44b708bccc2e2d5385cc0: ending tablet copy session 9db17f84d37944098cbeb9e0d8456efb-ca458907378e42d48adaa0ee647db4cd on tablet ca458907378e42d48adaa0ee647db4cd with peer 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:55:53.288233 10044 tablet_copy_client.cc:308] T 086509f1ee0a4a81b2f8194b50dbe159 P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.130:38197
I20240502 01:55:53.288419  6467 consensus_queue.cc:237] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 01:55:53.290505  6224 tablet_copy_service.cc:140] P a2c90ee440b44b708bccc2e2d5385cc0: Received BeginTabletCopySession request for tablet 086509f1ee0a4a81b2f8194b50dbe159 from peer 436822a397664c54b8d1b9eb3bc6cff4 ({username='slave'} at 127.25.109.135:34595)
I20240502 01:55:53.290732  6224 tablet_copy_service.cc:161] P a2c90ee440b44b708bccc2e2d5385cc0: Beginning new tablet copy session on tablet 086509f1ee0a4a81b2f8194b50dbe159 from peer 436822a397664c54b8d1b9eb3bc6cff4 at {username='slave'} at 127.25.109.135:34595: session id = 436822a397664c54b8d1b9eb3bc6cff4-086509f1ee0a4a81b2f8194b50dbe159
I20240502 01:55:53.295148  6224 tablet_copy_source_session.cc:205] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:53.296821  8406 raft_consensus.cc:1261] T 6984d638991442e3adaed1b24bee95e9 P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:53.296995 10044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 086509f1ee0a4a81b2f8194b50dbe159. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:53.297626  9317 consensus_queue.cc:1035] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:53.301548  6067 raft_consensus.cc:1261] T 6984d638991442e3adaed1b24bee95e9 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 01:55:53.302305  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133:43379): Couldn't send request to peer 60298ffea1dd4c25a38765c2ec9ec9d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6984d638991442e3adaed1b24bee95e9. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:53.303051  9317 consensus_queue.cc:1035] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:53.305604 10044 tablet_copy_client.cc:783] T 086509f1ee0a4a81b2f8194b50dbe159 P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:53.305897 10044 tablet_copy_client.cc:647] T 086509f1ee0a4a81b2f8194b50dbe159 P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:53.308490 10044 tablet_copy_client.cc:523] T 086509f1ee0a4a81b2f8194b50dbe159 P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:53.308591  6466 consensus_queue.cc:237] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:55:53.313293 10044 tablet_bootstrap.cc:492] T 086509f1ee0a4a81b2f8194b50dbe159 P 436822a397664c54b8d1b9eb3bc6cff4: Bootstrap starting.
I20240502 01:55:53.314312  6336 raft_consensus.cc:1261] T 6984d638991442e3adaed1b24bee95e9 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:53.325351 10064 consensus_queue.cc:1035] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 01:55:53.327617  6268 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135:40543): Couldn't send request to peer 436822a397664c54b8d1b9eb3bc6cff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: afa5561f8c0e4c0fa9bbbd5da61348c7. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:53.329017  6066 raft_consensus.cc:1261] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:53.329788  9705 consensus_queue.cc:1035] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:53.328578  9317 raft_consensus.cc:2932] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:55:53.334772  6467 raft_consensus.cc:1261] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:53.335487  9395 consensus_queue.cc:1035] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:53.339434  6068 consensus_queue.cc:237] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:55:53.343464  8405 raft_consensus.cc:2932] T 6984d638991442e3adaed1b24bee95e9 P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:55:53.357095  6065 raft_consensus.cc:2932] T 6984d638991442e3adaed1b24bee95e9 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:55:53.358091  5920 catalog_manager.cc:5484] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 reported cstate change: config changed from index 4 to 5, VOTER 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133) added. New cstate: current_term: 1 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
W20240502 01:55:53.369405  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eb91f5df851459b8ecf09cbe34fefa5. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:53.371150  6201 raft_consensus.cc:1261] T 4eb91f5df851459b8ecf09cbe34fefa5 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:53.372059  9898 consensus_queue.cc:1035] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:53.374089  8135 raft_consensus.cc:1261] T 4eb91f5df851459b8ecf09cbe34fefa5 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:53.374082  6334 raft_consensus.cc:1261] T 4eb91f5df851459b8ecf09cbe34fefa5 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:53.375375  9898 consensus_queue.cc:1035] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:53.376477  9898 consensus_queue.cc:1035] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:53.394402  6336 raft_consensus.cc:2932] T 6984d638991442e3adaed1b24bee95e9 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
W20240502 01:55:53.404520  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5be45a68cff24a75a03b2fcd4b7851f9. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:53.410180 10064 raft_consensus.cc:2932] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:53.411953  9395 raft_consensus.cc:2932] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:55:53.416033  6201 raft_consensus.cc:2932] T 4eb91f5df851459b8ecf09cbe34fefa5 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:53.416460  6066 raft_consensus.cc:2932] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:55:53.422360  5916 catalog_manager.cc:5484] T 4eb91f5df851459b8ecf09cbe34fefa5 P a2c90ee440b44b708bccc2e2d5385cc0 reported cstate change: config changed from index 4 to 5, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New cstate: current_term: 1 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:53.426142  8135 raft_consensus.cc:2932] T 4eb91f5df851459b8ecf09cbe34fefa5 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:53.429556 10083 ts_tablet_manager.cc:902] T 74eadf4c939c4c6383732cd74c5ec9d6 P 60298ffea1dd4c25a38765c2ec9ec9d5: Initiating tablet copy from peer 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245)
I20240502 01:55:53.431183  6467 raft_consensus.cc:2932] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:55:53.431548  6335 raft_consensus.cc:1261] T 5be45a68cff24a75a03b2fcd4b7851f9 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:53.431571  6201 raft_consensus.cc:1261] T 5be45a68cff24a75a03b2fcd4b7851f9 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:53.432425  6334 raft_consensus.cc:2932] T 4eb91f5df851459b8ecf09cbe34fefa5 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:53.433081  8951 consensus_queue.cc:1035] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:53.434532  8951 consensus_queue.cc:1035] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:53.445601  5912 catalog_manager.cc:5484] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 reported cstate change: config changed from index -1 to 4, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:55:53.505843 10044 tablet_bootstrap.cc:492] T 086509f1ee0a4a81b2f8194b50dbe159 P 436822a397664c54b8d1b9eb3bc6cff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:53.506294 10044 tablet_bootstrap.cc:492] T 086509f1ee0a4a81b2f8194b50dbe159 P 436822a397664c54b8d1b9eb3bc6cff4: Bootstrap complete.
I20240502 01:55:53.506613 10044 ts_tablet_manager.cc:1367] T 086509f1ee0a4a81b2f8194b50dbe159 P 436822a397664c54b8d1b9eb3bc6cff4: Time spent bootstrapping tablet: real 0.193s	user 0.039s	sys 0.023s
I20240502 01:55:53.507515 10044 raft_consensus.cc:348] T 086509f1ee0a4a81b2f8194b50dbe159 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:55:53.507798 10044 raft_consensus.cc:729] T 086509f1ee0a4a81b2f8194b50dbe159 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 436822a397664c54b8d1b9eb3bc6cff4, State: Initialized, Role: FOLLOWER
I20240502 01:55:53.508101 10044 consensus_queue.cc:260] T 086509f1ee0a4a81b2f8194b50dbe159 P 436822a397664c54b8d1b9eb3bc6cff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:55:53.509442 10003 tablet_bootstrap.cc:492] T 73ed91df7ff04846991c95b30a86ea7e P 436822a397664c54b8d1b9eb3bc6cff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:53.509953 10003 tablet_bootstrap.cc:492] T 73ed91df7ff04846991c95b30a86ea7e P 436822a397664c54b8d1b9eb3bc6cff4: Bootstrap complete.
I20240502 01:55:53.514106 10003 ts_tablet_manager.cc:1367] T 73ed91df7ff04846991c95b30a86ea7e P 436822a397664c54b8d1b9eb3bc6cff4: Time spent bootstrapping tablet: real 0.279s	user 0.064s	sys 0.023s
I20240502 01:55:53.517546 10044 ts_tablet_manager.cc:1398] T 086509f1ee0a4a81b2f8194b50dbe159 P 436822a397664c54b8d1b9eb3bc6cff4: Time spent starting tablet: real 0.011s	user 0.003s	sys 0.008s
I20240502 01:55:53.518321  6224 tablet_copy_service.cc:342] P a2c90ee440b44b708bccc2e2d5385cc0: Request end of tablet copy session 436822a397664c54b8d1b9eb3bc6cff4-086509f1ee0a4a81b2f8194b50dbe159 received from {username='slave'} at 127.25.109.135:34595
I20240502 01:55:53.518445  6224 tablet_copy_service.cc:433] P a2c90ee440b44b708bccc2e2d5385cc0: ending tablet copy session 436822a397664c54b8d1b9eb3bc6cff4-086509f1ee0a4a81b2f8194b50dbe159 on tablet 086509f1ee0a4a81b2f8194b50dbe159 with peer 436822a397664c54b8d1b9eb3bc6cff4
I20240502 01:55:53.518311  8951 raft_consensus.cc:2932] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:53.526468  6201 raft_consensus.cc:2932] T 5be45a68cff24a75a03b2fcd4b7851f9 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:53.535430 10003 raft_consensus.cc:348] T 73ed91df7ff04846991c95b30a86ea7e P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:55:53.535727 10003 raft_consensus.cc:729] T 73ed91df7ff04846991c95b30a86ea7e P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 436822a397664c54b8d1b9eb3bc6cff4, State: Initialized, Role: FOLLOWER
I20240502 01:55:53.536002 10003 consensus_queue.cc:260] T 73ed91df7ff04846991c95b30a86ea7e P 436822a397664c54b8d1b9eb3bc6cff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:55:53.536617 10003 ts_tablet_manager.cc:1398] T 73ed91df7ff04846991c95b30a86ea7e P 436822a397664c54b8d1b9eb3bc6cff4: Time spent starting tablet: real 0.022s	user 0.002s	sys 0.000s
I20240502 01:55:53.536537  5920 catalog_manager.cc:5484] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 reported cstate change: config changed from index -1 to 4, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New cstate: current_term: 1 leader_uuid: "218bc8c1f9ce46059898fc738179adf9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:53.541070  6489 tablet_copy_service.cc:342] P 07cdb213c77c4ffda1f999b66b725399: Request end of tablet copy session 436822a397664c54b8d1b9eb3bc6cff4-73ed91df7ff04846991c95b30a86ea7e received from {username='slave'} at 127.25.109.135:41247
I20240502 01:55:53.541217  6489 tablet_copy_service.cc:433] P 07cdb213c77c4ffda1f999b66b725399: ending tablet copy session 436822a397664c54b8d1b9eb3bc6cff4-73ed91df7ff04846991c95b30a86ea7e on tablet 73ed91df7ff04846991c95b30a86ea7e with peer 436822a397664c54b8d1b9eb3bc6cff4
I20240502 01:55:53.541028  6334 raft_consensus.cc:2932] T 5be45a68cff24a75a03b2fcd4b7851f9 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:53.579672  8135 raft_consensus.cc:1203] T ca458907378e42d48adaa0ee647db4cd P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:55:53.584977 10083 tablet_copy_client.cc:308] T 74eadf4c939c4c6383732cd74c5ec9d6 P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.131:44245
I20240502 01:55:53.586370  6357 tablet_copy_service.cc:140] P 94ffd7aaf85b4e17881241b8396fa2f3: Received BeginTabletCopySession request for tablet 74eadf4c939c4c6383732cd74c5ec9d6 from peer 60298ffea1dd4c25a38765c2ec9ec9d5 ({username='slave'} at 127.25.109.133:43975)
I20240502 01:55:53.586563  6357 tablet_copy_service.cc:161] P 94ffd7aaf85b4e17881241b8396fa2f3: Beginning new tablet copy session on tablet 74eadf4c939c4c6383732cd74c5ec9d6 from peer 60298ffea1dd4c25a38765c2ec9ec9d5 at {username='slave'} at 127.25.109.133:43975: session id = 60298ffea1dd4c25a38765c2ec9ec9d5-74eadf4c939c4c6383732cd74c5ec9d6
I20240502 01:55:53.591219  6357 tablet_copy_source_session.cc:205] T 74eadf4c939c4c6383732cd74c5ec9d6 P 94ffd7aaf85b4e17881241b8396fa2f3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:53.594238 10083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74eadf4c939c4c6383732cd74c5ec9d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:53.602079 10083 tablet_copy_client.cc:783] T 74eadf4c939c4c6383732cd74c5ec9d6 P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:53.602368 10083 tablet_copy_client.cc:647] T 74eadf4c939c4c6383732cd74c5ec9d6 P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:53.604384 10083 tablet_copy_client.cc:523] T 74eadf4c939c4c6383732cd74c5ec9d6 P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:53.607795 10083 tablet_bootstrap.cc:492] T 74eadf4c939c4c6383732cd74c5ec9d6 P 60298ffea1dd4c25a38765c2ec9ec9d5: Bootstrap starting.
I20240502 01:55:53.678988  6066 consensus_queue.cc:237] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:55:53.679369  8270 raft_consensus.cc:1203] T 086509f1ee0a4a81b2f8194b50dbe159 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:55:53.687491  8135 raft_consensus.cc:1261] T c58d6b2401c047cd9af56a66c2e00105 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:53.689364  6467 raft_consensus.cc:1261] T c58d6b2401c047cd9af56a66c2e00105 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:53.690768  6201 raft_consensus.cc:1261] T c58d6b2401c047cd9af56a66c2e00105 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:53.691735  8951 consensus_queue.cc:1035] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:53.692437  8951 consensus_queue.cc:1035] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:53.694638 10042 consensus_queue.cc:1035] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:53.695694 10083 tablet_bootstrap.cc:492] T 74eadf4c939c4c6383732cd74c5ec9d6 P 60298ffea1dd4c25a38765c2ec9ec9d5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:53.696926 10083 tablet_bootstrap.cc:492] T 74eadf4c939c4c6383732cd74c5ec9d6 P 60298ffea1dd4c25a38765c2ec9ec9d5: Bootstrap complete.
I20240502 01:55:53.697371 10083 ts_tablet_manager.cc:1367] T 74eadf4c939c4c6383732cd74c5ec9d6 P 60298ffea1dd4c25a38765c2ec9ec9d5: Time spent bootstrapping tablet: real 0.090s	user 0.045s	sys 0.008s
I20240502 01:55:53.699693 10083 raft_consensus.cc:348] T 74eadf4c939c4c6383732cd74c5ec9d6 P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 01:55:53.700013 10083 raft_consensus.cc:729] T 74eadf4c939c4c6383732cd74c5ec9d6 P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60298ffea1dd4c25a38765c2ec9ec9d5, State: Initialized, Role: FOLLOWER
I20240502 01:55:53.706648 10042 raft_consensus.cc:2932] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:53.709976 10003 ts_tablet_manager.cc:902] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4: Initiating tablet copy from peer 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245)
I20240502 01:55:53.711683  8135 raft_consensus.cc:2932] T c58d6b2401c047cd9af56a66c2e00105 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:53.714911 10083 consensus_queue.cc:260] T 74eadf4c939c4c6383732cd74c5ec9d6 P 60298ffea1dd4c25a38765c2ec9ec9d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 01:55:53.723749  5920 catalog_manager.cc:5484] T c58d6b2401c047cd9af56a66c2e00105 P 9db17f84d37944098cbeb9e0d8456efb reported cstate change: config changed from index 4 to 5, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New cstate: current_term: 1 leader_uuid: "218bc8c1f9ce46059898fc738179adf9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:53.730562  6201 raft_consensus.cc:2932] T c58d6b2401c047cd9af56a66c2e00105 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:53.761055  6467 raft_consensus.cc:2932] T c58d6b2401c047cd9af56a66c2e00105 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:53.771728 10083 ts_tablet_manager.cc:1398] T 74eadf4c939c4c6383732cd74c5ec9d6 P 60298ffea1dd4c25a38765c2ec9ec9d5: Time spent starting tablet: real 0.073s	user 0.013s	sys 0.017s
I20240502 01:55:53.773540  6357 tablet_copy_service.cc:342] P 94ffd7aaf85b4e17881241b8396fa2f3: Request end of tablet copy session 60298ffea1dd4c25a38765c2ec9ec9d5-74eadf4c939c4c6383732cd74c5ec9d6 received from {username='slave'} at 127.25.109.133:43975
I20240502 01:55:53.773715  6357 tablet_copy_service.cc:433] P 94ffd7aaf85b4e17881241b8396fa2f3: ending tablet copy session 60298ffea1dd4c25a38765c2ec9ec9d5-74eadf4c939c4c6383732cd74c5ec9d6 on tablet 74eadf4c939c4c6383732cd74c5ec9d6 with peer 60298ffea1dd4c25a38765c2ec9ec9d5
I20240502 01:55:53.776335 10003 tablet_copy_client.cc:308] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.131:44245
I20240502 01:55:53.777151  6357 tablet_copy_service.cc:140] P 94ffd7aaf85b4e17881241b8396fa2f3: Received BeginTabletCopySession request for tablet afa5561f8c0e4c0fa9bbbd5da61348c7 from peer 436822a397664c54b8d1b9eb3bc6cff4 ({username='slave'} at 127.25.109.135:37865)
I20240502 01:55:53.777338  6357 tablet_copy_service.cc:161] P 94ffd7aaf85b4e17881241b8396fa2f3: Beginning new tablet copy session on tablet afa5561f8c0e4c0fa9bbbd5da61348c7 from peer 436822a397664c54b8d1b9eb3bc6cff4 at {username='slave'} at 127.25.109.135:37865: session id = 436822a397664c54b8d1b9eb3bc6cff4-afa5561f8c0e4c0fa9bbbd5da61348c7
I20240502 01:55:53.777959  6467 consensus_queue.cc:237] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 01:55:53.781013  6357 tablet_copy_source_session.cc:205] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3: Tablet Copy: opened 0 blocks and 1 log segments
W20240502 01:55:53.782718  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c58d6b2401c047cd9af56a66c2e00105. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:53.784165 10003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afa5561f8c0e4c0fa9bbbd5da61348c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:53.788214 10083 ts_tablet_manager.cc:902] T 6984d638991442e3adaed1b24bee95e9 P 60298ffea1dd4c25a38765c2ec9ec9d5: Initiating tablet copy from peer 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867)
I20240502 01:55:53.790655 10003 tablet_copy_client.cc:783] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:53.790903 10003 tablet_copy_client.cc:647] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Starting download of 1 WAL segments...
W20240502 01:55:53.791553  6402 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133:43379): Couldn't send request to peer 60298ffea1dd4c25a38765c2ec9ec9d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 02b106ec57b94e44a29947a80a030a23. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:53.793452 10003 tablet_copy_client.cc:523] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:53.793789  6066 raft_consensus.cc:1261] T 02b106ec57b94e44a29947a80a030a23 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:53.794425  9317 consensus_queue.cc:1035] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:53.796974 10003 tablet_bootstrap.cc:492] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4: Bootstrap starting.
I20240502 01:55:53.807531  6201 raft_consensus.cc:1261] T 02b106ec57b94e44a29947a80a030a23 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:53.808277  9317 consensus_queue.cc:1035] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:53.823173  9317 raft_consensus.cc:2932] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:55:53.829274  6203 raft_consensus.cc:2932] T 02b106ec57b94e44a29947a80a030a23 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:55:53.838702  6334 consensus_queue.cc:237] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:55:53.843030  5922 catalog_manager.cc:5484] T 02b106ec57b94e44a29947a80a030a23 P a2c90ee440b44b708bccc2e2d5385cc0 reported cstate change: config changed from index -1 to 4, VOTER 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133) added. New cstate: current_term: 1 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:55:53.854147 10083 tablet_copy_client.cc:308] T 6984d638991442e3adaed1b24bee95e9 P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.132:36867
I20240502 01:55:53.854908  6489 tablet_copy_service.cc:140] P 07cdb213c77c4ffda1f999b66b725399: Received BeginTabletCopySession request for tablet 6984d638991442e3adaed1b24bee95e9 from peer 60298ffea1dd4c25a38765c2ec9ec9d5 ({username='slave'} at 127.25.109.133:42089)
I20240502 01:55:53.855108  6489 tablet_copy_service.cc:161] P 07cdb213c77c4ffda1f999b66b725399: Beginning new tablet copy session on tablet 6984d638991442e3adaed1b24bee95e9 from peer 60298ffea1dd4c25a38765c2ec9ec9d5 at {username='slave'} at 127.25.109.133:42089: session id = 60298ffea1dd4c25a38765c2ec9ec9d5-6984d638991442e3adaed1b24bee95e9
I20240502 01:55:53.858259  6489 tablet_copy_source_session.cc:205] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:53.860045 10083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6984d638991442e3adaed1b24bee95e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:53.864605  6203 consensus_queue.cc:237] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
W20240502 01:55:53.867048  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2cefdfa8851147999c1425c606f36b1f. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:53.867101  6204 raft_consensus.cc:1261] T 2cefdfa8851147999c1425c606f36b1f P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:53.868257  9395 consensus_queue.cc:1035] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:53.870891  6467 raft_consensus.cc:1261] T 2cefdfa8851147999c1425c606f36b1f P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:53.872486  9705 consensus_queue.cc:1035] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:53.875406  6066 raft_consensus.cc:2932] T 02b106ec57b94e44a29947a80a030a23 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:55:53.878542  8135 raft_consensus.cc:1261] T ca458907378e42d48adaa0ee647db4cd P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Refusing update from remote peer a2c90ee440b44b708bccc2e2d5385cc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:53.879187  6466 raft_consensus.cc:1261] T ca458907378e42d48adaa0ee647db4cd P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Refusing update from remote peer a2c90ee440b44b708bccc2e2d5385cc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:53.879261  6068 raft_consensus.cc:1261] T ca458907378e42d48adaa0ee647db4cd P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Refusing update from remote peer a2c90ee440b44b708bccc2e2d5385cc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:53.879961  9945 consensus_queue.cc:1035] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:53.880486  9945 consensus_queue.cc:1035] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:53.880934 10193 consensus_queue.cc:1035] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 01:55:53.888021  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ca458907378e42d48adaa0ee647db4cd. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:53.889173 10083 tablet_copy_client.cc:783] T 6984d638991442e3adaed1b24bee95e9 P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:53.889688 10083 tablet_copy_client.cc:647] T 6984d638991442e3adaed1b24bee95e9 P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:53.892181 10083 tablet_copy_client.cc:523] T 6984d638991442e3adaed1b24bee95e9 P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:53.892537  8001 raft_consensus.cc:1203] T 74eadf4c939c4c6383732cd74c5ec9d6 P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:55:53.903056 10083 tablet_bootstrap.cc:492] T 6984d638991442e3adaed1b24bee95e9 P 60298ffea1dd4c25a38765c2ec9ec9d5: Bootstrap starting.
I20240502 01:55:53.906533 10003 tablet_bootstrap.cc:492] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:53.907094 10003 tablet_bootstrap.cc:492] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4: Bootstrap complete.
I20240502 01:55:53.907414 10003 ts_tablet_manager.cc:1367] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4: Time spent bootstrapping tablet: real 0.111s	user 0.044s	sys 0.011s
I20240502 01:55:53.907658 10197 ts_tablet_manager.cc:902] T 4eb91f5df851459b8ecf09cbe34fefa5 P 6d34e50fb5dd4a1db58013d4863b8992: Initiating tablet copy from peer 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867)
I20240502 01:55:53.908347 10003 raft_consensus.cc:348] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:55:53.908624 10003 raft_consensus.cc:729] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 436822a397664c54b8d1b9eb3bc6cff4, State: Initialized, Role: FOLLOWER
I20240502 01:55:53.908918 10003 consensus_queue.cc:260] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:55:53.909520 10003 ts_tablet_manager.cc:1398] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240502 01:55:53.912936  6357 tablet_copy_service.cc:342] P 94ffd7aaf85b4e17881241b8396fa2f3: Request end of tablet copy session 436822a397664c54b8d1b9eb3bc6cff4-afa5561f8c0e4c0fa9bbbd5da61348c7 received from {username='slave'} at 127.25.109.135:37865
I20240502 01:55:53.913092  6357 tablet_copy_service.cc:433] P 94ffd7aaf85b4e17881241b8396fa2f3: ending tablet copy session 436822a397664c54b8d1b9eb3bc6cff4-afa5561f8c0e4c0fa9bbbd5da61348c7 on tablet afa5561f8c0e4c0fa9bbbd5da61348c7 with peer 436822a397664c54b8d1b9eb3bc6cff4
I20240502 01:55:53.917145  9945 raft_consensus.cc:2932] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:53.917905  8135 raft_consensus.cc:2932] T ca458907378e42d48adaa0ee647db4cd P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:53.924088  5922 catalog_manager.cc:5484] T ca458907378e42d48adaa0ee647db4cd P 9db17f84d37944098cbeb9e0d8456efb reported cstate change: config changed from index 4 to 5, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New cstate: current_term: 1 leader_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:53.926446  6466 raft_consensus.cc:2932] T ca458907378e42d48adaa0ee647db4cd P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:53.947389  6068 raft_consensus.cc:2932] T ca458907378e42d48adaa0ee647db4cd P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:53.950770  9705 raft_consensus.cc:2932] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:53.965507  8270 raft_consensus.cc:1203] T 73ed91df7ff04846991c95b30a86ea7e P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 01:55:53.967047  6201 raft_consensus.cc:2932] T 2cefdfa8851147999c1425c606f36b1f P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:53.967350  6467 raft_consensus.cc:2932] T 2cefdfa8851147999c1425c606f36b1f P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:53.979007  5920 catalog_manager.cc:5484] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 reported cstate change: config changed from index -1 to 4, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:53.984829 10197 tablet_copy_client.cc:308] T 4eb91f5df851459b8ecf09cbe34fefa5 P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.132:36867
I20240502 01:55:53.985745  6489 tablet_copy_service.cc:140] P 07cdb213c77c4ffda1f999b66b725399: Received BeginTabletCopySession request for tablet 4eb91f5df851459b8ecf09cbe34fefa5 from peer 6d34e50fb5dd4a1db58013d4863b8992 ({username='slave'} at 127.25.109.136:34855)
I20240502 01:55:53.985942  6489 tablet_copy_service.cc:161] P 07cdb213c77c4ffda1f999b66b725399: Beginning new tablet copy session on tablet 4eb91f5df851459b8ecf09cbe34fefa5 from peer 6d34e50fb5dd4a1db58013d4863b8992 at {username='slave'} at 127.25.109.136:34855: session id = 6d34e50fb5dd4a1db58013d4863b8992-4eb91f5df851459b8ecf09cbe34fefa5
I20240502 01:55:53.988873  6489 tablet_copy_source_session.cc:205] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:53.993945 10197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4eb91f5df851459b8ecf09cbe34fefa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:54.003294 10197 tablet_copy_client.cc:783] T 4eb91f5df851459b8ecf09cbe34fefa5 P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:54.003582 10197 tablet_copy_client.cc:647] T 4eb91f5df851459b8ecf09cbe34fefa5 P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:54.005995 10197 tablet_copy_client.cc:523] T 4eb91f5df851459b8ecf09cbe34fefa5 P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:54.010645 10197 tablet_bootstrap.cc:492] T 4eb91f5df851459b8ecf09cbe34fefa5 P 6d34e50fb5dd4a1db58013d4863b8992: Bootstrap starting.
I20240502 01:55:54.024287 10215 ts_tablet_manager.cc:902] T 5be45a68cff24a75a03b2fcd4b7851f9 P 9db17f84d37944098cbeb9e0d8456efb: Initiating tablet copy from peer 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227)
I20240502 01:55:54.104007 10197 tablet_bootstrap.cc:492] T 4eb91f5df851459b8ecf09cbe34fefa5 P 6d34e50fb5dd4a1db58013d4863b8992: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:54.104655 10197 tablet_bootstrap.cc:492] T 4eb91f5df851459b8ecf09cbe34fefa5 P 6d34e50fb5dd4a1db58013d4863b8992: Bootstrap complete.
I20240502 01:55:54.104980 10197 ts_tablet_manager.cc:1367] T 4eb91f5df851459b8ecf09cbe34fefa5 P 6d34e50fb5dd4a1db58013d4863b8992: Time spent bootstrapping tablet: real 0.094s	user 0.061s	sys 0.013s
I20240502 01:55:54.105996 10197 raft_consensus.cc:348] T 4eb91f5df851459b8ecf09cbe34fefa5 P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:55:54.106544 10197 raft_consensus.cc:729] T 4eb91f5df851459b8ecf09cbe34fefa5 P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d34e50fb5dd4a1db58013d4863b8992, State: Initialized, Role: FOLLOWER
I20240502 01:55:54.106849 10197 consensus_queue.cc:260] T 4eb91f5df851459b8ecf09cbe34fefa5 P 6d34e50fb5dd4a1db58013d4863b8992 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:55:54.114838 10215 tablet_copy_client.cc:308] T 5be45a68cff24a75a03b2fcd4b7851f9 P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.129:36227
I20240502 01:55:54.116456  6089 tablet_copy_service.cc:140] P 218bc8c1f9ce46059898fc738179adf9: Received BeginTabletCopySession request for tablet 5be45a68cff24a75a03b2fcd4b7851f9 from peer 9db17f84d37944098cbeb9e0d8456efb ({username='slave'} at 127.25.109.134:39709)
I20240502 01:55:54.116639  6089 tablet_copy_service.cc:161] P 218bc8c1f9ce46059898fc738179adf9: Beginning new tablet copy session on tablet 5be45a68cff24a75a03b2fcd4b7851f9 from peer 9db17f84d37944098cbeb9e0d8456efb at {username='slave'} at 127.25.109.134:39709: session id = 9db17f84d37944098cbeb9e0d8456efb-5be45a68cff24a75a03b2fcd4b7851f9
I20240502 01:55:54.119632  6089 tablet_copy_source_session.cc:205] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:54.122140 10215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5be45a68cff24a75a03b2fcd4b7851f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:54.138134  6467 consensus_queue.cc:237] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:55:54.140480 10197 ts_tablet_manager.cc:1398] T 4eb91f5df851459b8ecf09cbe34fefa5 P 6d34e50fb5dd4a1db58013d4863b8992: Time spent starting tablet: real 0.035s	user 0.010s	sys 0.015s
I20240502 01:55:54.142074  6489 tablet_copy_service.cc:342] P 07cdb213c77c4ffda1f999b66b725399: Request end of tablet copy session 6d34e50fb5dd4a1db58013d4863b8992-4eb91f5df851459b8ecf09cbe34fefa5 received from {username='slave'} at 127.25.109.136:34855
I20240502 01:55:54.142253  6489 tablet_copy_service.cc:433] P 07cdb213c77c4ffda1f999b66b725399: ending tablet copy session 6d34e50fb5dd4a1db58013d4863b8992-4eb91f5df851459b8ecf09cbe34fefa5 on tablet 4eb91f5df851459b8ecf09cbe34fefa5 with peer 6d34e50fb5dd4a1db58013d4863b8992
I20240502 01:55:54.144975 10197 ts_tablet_manager.cc:902] T c58d6b2401c047cd9af56a66c2e00105 P 6d34e50fb5dd4a1db58013d4863b8992: Initiating tablet copy from peer 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227)
I20240502 01:55:54.147007 10215 tablet_copy_client.cc:783] T 5be45a68cff24a75a03b2fcd4b7851f9 P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:54.148593 10215 tablet_copy_client.cc:647] T 5be45a68cff24a75a03b2fcd4b7851f9 P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:54.153671 10215 tablet_copy_client.cc:523] T 5be45a68cff24a75a03b2fcd4b7851f9 P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:54.157449 10215 tablet_bootstrap.cc:492] T 5be45a68cff24a75a03b2fcd4b7851f9 P 9db17f84d37944098cbeb9e0d8456efb: Bootstrap starting.
W20240502 01:55:54.163272  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f309b40931dd444091869119eab33bbc. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:54.169054  6068 consensus_queue.cc:237] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:55:54.169622  6204 raft_consensus.cc:1261] T f309b40931dd444091869119eab33bbc P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:54.170352  9317 consensus_queue.cc:1035] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:55:54.172518  5998 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135:40543): Couldn't send request to peer 436822a397664c54b8d1b9eb3bc6cff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5be45a68cff24a75a03b2fcd4b7851f9. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:54.173586  6334 raft_consensus.cc:1261] T 5be45a68cff24a75a03b2fcd4b7851f9 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 01:55:54.175455  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:54.176084 10145 consensus_queue.cc:1035] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:54.177482  6201 raft_consensus.cc:1261] T 5be45a68cff24a75a03b2fcd4b7851f9 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:54.178217 10145 consensus_queue.cc:1035] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:54.184113 10145 raft_consensus.cc:2932] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:55:54.188508  6334 raft_consensus.cc:2932] T 5be45a68cff24a75a03b2fcd4b7851f9 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:55:54.203279  5920 catalog_manager.cc:5484] T 5be45a68cff24a75a03b2fcd4b7851f9 P 94ffd7aaf85b4e17881241b8396fa2f3 reported cstate change: config changed from index 4 to 5, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New cstate: current_term: 1 leader_uuid: "218bc8c1f9ce46059898fc738179adf9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:55:54.204735 10083 tablet_bootstrap.cc:492] T 6984d638991442e3adaed1b24bee95e9 P 60298ffea1dd4c25a38765c2ec9ec9d5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:54.205217 10083 tablet_bootstrap.cc:492] T 6984d638991442e3adaed1b24bee95e9 P 60298ffea1dd4c25a38765c2ec9ec9d5: Bootstrap complete.
I20240502 01:55:54.205518 10083 ts_tablet_manager.cc:1367] T 6984d638991442e3adaed1b24bee95e9 P 60298ffea1dd4c25a38765c2ec9ec9d5: Time spent bootstrapping tablet: real 0.303s	user 0.047s	sys 0.003s
I20240502 01:55:54.205695  6335 raft_consensus.cc:1261] T f309b40931dd444091869119eab33bbc P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:54.206321 10083 raft_consensus.cc:348] T 6984d638991442e3adaed1b24bee95e9 P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 01:55:54.206568 10083 raft_consensus.cc:729] T 6984d638991442e3adaed1b24bee95e9 P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60298ffea1dd4c25a38765c2ec9ec9d5, State: Initialized, Role: FOLLOWER
I20240502 01:55:54.206928 10083 consensus_queue.cc:260] T 6984d638991442e3adaed1b24bee95e9 P 60298ffea1dd4c25a38765c2ec9ec9d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 01:55:54.207551 10083 ts_tablet_manager.cc:1398] T 6984d638991442e3adaed1b24bee95e9 P 60298ffea1dd4c25a38765c2ec9ec9d5: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 01:55:54.208201  9317 consensus_queue.cc:1035] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:54.208792  6489 tablet_copy_service.cc:342] P 07cdb213c77c4ffda1f999b66b725399: Request end of tablet copy session 60298ffea1dd4c25a38765c2ec9ec9d5-6984d638991442e3adaed1b24bee95e9 received from {username='slave'} at 127.25.109.133:42089
I20240502 01:55:54.208949  6489 tablet_copy_service.cc:433] P 07cdb213c77c4ffda1f999b66b725399: ending tablet copy session 60298ffea1dd4c25a38765c2ec9ec9d5-6984d638991442e3adaed1b24bee95e9 on tablet 6984d638991442e3adaed1b24bee95e9 with peer 60298ffea1dd4c25a38765c2ec9ec9d5
I20240502 01:55:54.210492  6201 raft_consensus.cc:2932] T 5be45a68cff24a75a03b2fcd4b7851f9 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:55:54.234289  9317 raft_consensus.cc:2932] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:54.238566 10197 tablet_copy_client.cc:308] T c58d6b2401c047cd9af56a66c2e00105 P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.129:36227
I20240502 01:55:54.238943  6201 raft_consensus.cc:2932] T f309b40931dd444091869119eab33bbc P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:54.247148  5922 catalog_manager.cc:5484] T f309b40931dd444091869119eab33bbc P a2c90ee440b44b708bccc2e2d5385cc0 reported cstate change: config changed from index -1 to 4, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New cstate: current_term: 1 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:54.255019  6334 raft_consensus.cc:2932] T f309b40931dd444091869119eab33bbc P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:54.262964 10215 tablet_bootstrap.cc:492] T 5be45a68cff24a75a03b2fcd4b7851f9 P 9db17f84d37944098cbeb9e0d8456efb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:54.263522 10215 tablet_bootstrap.cc:492] T 5be45a68cff24a75a03b2fcd4b7851f9 P 9db17f84d37944098cbeb9e0d8456efb: Bootstrap complete.
I20240502 01:55:54.263916 10215 ts_tablet_manager.cc:1367] T 5be45a68cff24a75a03b2fcd4b7851f9 P 9db17f84d37944098cbeb9e0d8456efb: Time spent bootstrapping tablet: real 0.107s	user 0.043s	sys 0.010s
I20240502 01:55:54.264993 10215 raft_consensus.cc:348] T 5be45a68cff24a75a03b2fcd4b7851f9 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:55:54.265372 10215 raft_consensus.cc:729] T 5be45a68cff24a75a03b2fcd4b7851f9 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9db17f84d37944098cbeb9e0d8456efb, State: Initialized, Role: FOLLOWER
I20240502 01:55:54.265725 10215 consensus_queue.cc:260] T 5be45a68cff24a75a03b2fcd4b7851f9 P 9db17f84d37944098cbeb9e0d8456efb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:55:54.266549 10215 ts_tablet_manager.cc:1398] T 5be45a68cff24a75a03b2fcd4b7851f9 P 9db17f84d37944098cbeb9e0d8456efb: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 01:55:54.267916  6089 tablet_copy_service.cc:342] P 218bc8c1f9ce46059898fc738179adf9: Request end of tablet copy session 9db17f84d37944098cbeb9e0d8456efb-5be45a68cff24a75a03b2fcd4b7851f9 received from {username='slave'} at 127.25.109.134:39709
I20240502 01:55:54.268097  6089 tablet_copy_service.cc:433] P 218bc8c1f9ce46059898fc738179adf9: ending tablet copy session 9db17f84d37944098cbeb9e0d8456efb-5be45a68cff24a75a03b2fcd4b7851f9 on tablet 5be45a68cff24a75a03b2fcd4b7851f9 with peer 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:55:54.273514  6089 tablet_copy_service.cc:140] P 218bc8c1f9ce46059898fc738179adf9: Received BeginTabletCopySession request for tablet c58d6b2401c047cd9af56a66c2e00105 from peer 6d34e50fb5dd4a1db58013d4863b8992 ({username='slave'} at 127.25.109.136:45167)
I20240502 01:55:54.273741  6089 tablet_copy_service.cc:161] P 218bc8c1f9ce46059898fc738179adf9: Beginning new tablet copy session on tablet c58d6b2401c047cd9af56a66c2e00105 from peer 6d34e50fb5dd4a1db58013d4863b8992 at {username='slave'} at 127.25.109.136:45167: session id = 6d34e50fb5dd4a1db58013d4863b8992-c58d6b2401c047cd9af56a66c2e00105
I20240502 01:55:54.279924  6089 tablet_copy_source_session.cc:205] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:54.280843 10083 ts_tablet_manager.cc:902] T 02b106ec57b94e44a29947a80a030a23 P 60298ffea1dd4c25a38765c2ec9ec9d5: Initiating tablet copy from peer 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867)
I20240502 01:55:54.281647 10197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c58d6b2401c047cd9af56a66c2e00105. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:54.282312 10083 tablet_copy_client.cc:308] T 02b106ec57b94e44a29947a80a030a23 P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.132:36867
I20240502 01:55:54.283038  6489 tablet_copy_service.cc:140] P 07cdb213c77c4ffda1f999b66b725399: Received BeginTabletCopySession request for tablet 02b106ec57b94e44a29947a80a030a23 from peer 60298ffea1dd4c25a38765c2ec9ec9d5 ({username='slave'} at 127.25.109.133:42089)
I20240502 01:55:54.283239  6489 tablet_copy_service.cc:161] P 07cdb213c77c4ffda1f999b66b725399: Beginning new tablet copy session on tablet 02b106ec57b94e44a29947a80a030a23 from peer 60298ffea1dd4c25a38765c2ec9ec9d5 at {username='slave'} at 127.25.109.133:42089: session id = 60298ffea1dd4c25a38765c2ec9ec9d5-02b106ec57b94e44a29947a80a030a23
I20240502 01:55:54.286520  6489 tablet_copy_source_session.cc:205] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:54.289427 10083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02b106ec57b94e44a29947a80a030a23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:54.296867 10083 tablet_copy_client.cc:783] T 02b106ec57b94e44a29947a80a030a23 P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:54.297155 10083 tablet_copy_client.cc:647] T 02b106ec57b94e44a29947a80a030a23 P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:54.300503 10083 tablet_copy_client.cc:523] T 02b106ec57b94e44a29947a80a030a23 P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:54.300549  6334 consensus_queue.cc:237] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:55:54.304045 10083 tablet_bootstrap.cc:492] T 02b106ec57b94e44a29947a80a030a23 P 60298ffea1dd4c25a38765c2ec9ec9d5: Bootstrap starting.
I20240502 01:55:54.305488  6467 raft_consensus.cc:1261] T 2cefdfa8851147999c1425c606f36b1f P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 01:55:54.307130  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2cefdfa8851147999c1425c606f36b1f. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:54.308327  8270 raft_consensus.cc:1203] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:55:54.310487  9395 consensus_queue.cc:1035] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 01:55:54.311468  6268 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135:40543): Couldn't send request to peer 436822a397664c54b8d1b9eb3bc6cff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2cefdfa8851147999c1425c606f36b1f. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:54.316991 10197 tablet_copy_client.cc:783] T c58d6b2401c047cd9af56a66c2e00105 P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:54.334571 10197 tablet_copy_client.cc:647] T c58d6b2401c047cd9af56a66c2e00105 P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:54.338533 10197 tablet_copy_client.cc:523] T c58d6b2401c047cd9af56a66c2e00105 P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:54.338984  8405 raft_consensus.cc:1203] T 4eb91f5df851459b8ecf09cbe34fefa5 P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 01:55:54.343101 10197 tablet_bootstrap.cc:492] T c58d6b2401c047cd9af56a66c2e00105 P 6d34e50fb5dd4a1db58013d4863b8992: Bootstrap starting.
I20240502 01:55:54.350970  6204 raft_consensus.cc:1261] T 2cefdfa8851147999c1425c606f36b1f P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:54.351837  9395 consensus_queue.cc:1035] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:54.355134  9395 raft_consensus.cc:2932] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:55:54.357568  6204 raft_consensus.cc:2932] T 2cefdfa8851147999c1425c606f36b1f P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:55:54.366950  5920 catalog_manager.cc:5484] T 2cefdfa8851147999c1425c606f36b1f P a2c90ee440b44b708bccc2e2d5385cc0 reported cstate change: config changed from index 4 to 5, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:55:54.369184  6467 raft_consensus.cc:2932] T 2cefdfa8851147999c1425c606f36b1f P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:55:54.396315  8001 raft_consensus.cc:1203] T 6984d638991442e3adaed1b24bee95e9 P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 01:55:54.476058 10255 ts_tablet_manager.cc:902] T ca458907378e42d48adaa0ee647db4cd P 6d34e50fb5dd4a1db58013d4863b8992: Initiating tablet copy from peer a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197)
I20240502 01:55:54.488727 10197 tablet_bootstrap.cc:492] T c58d6b2401c047cd9af56a66c2e00105 P 6d34e50fb5dd4a1db58013d4863b8992: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:54.489280 10197 tablet_bootstrap.cc:492] T c58d6b2401c047cd9af56a66c2e00105 P 6d34e50fb5dd4a1db58013d4863b8992: Bootstrap complete.
I20240502 01:55:54.489579 10197 ts_tablet_manager.cc:1367] T c58d6b2401c047cd9af56a66c2e00105 P 6d34e50fb5dd4a1db58013d4863b8992: Time spent bootstrapping tablet: real 0.147s	user 0.052s	sys 0.001s
I20240502 01:55:54.493858 10197 raft_consensus.cc:348] T c58d6b2401c047cd9af56a66c2e00105 P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:55:54.494151 10197 raft_consensus.cc:729] T c58d6b2401c047cd9af56a66c2e00105 P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d34e50fb5dd4a1db58013d4863b8992, State: Initialized, Role: FOLLOWER
I20240502 01:55:54.494438 10197 consensus_queue.cc:260] T c58d6b2401c047cd9af56a66c2e00105 P 6d34e50fb5dd4a1db58013d4863b8992 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:55:54.499372 10197 ts_tablet_manager.cc:1398] T c58d6b2401c047cd9af56a66c2e00105 P 6d34e50fb5dd4a1db58013d4863b8992: Time spent starting tablet: real 0.010s	user 0.003s	sys 0.000s
I20240502 01:55:54.500277  6089 tablet_copy_service.cc:342] P 218bc8c1f9ce46059898fc738179adf9: Request end of tablet copy session 6d34e50fb5dd4a1db58013d4863b8992-c58d6b2401c047cd9af56a66c2e00105 received from {username='slave'} at 127.25.109.136:45167
I20240502 01:55:54.500456  6089 tablet_copy_service.cc:433] P 218bc8c1f9ce46059898fc738179adf9: ending tablet copy session 6d34e50fb5dd4a1db58013d4863b8992-c58d6b2401c047cd9af56a66c2e00105 on tablet c58d6b2401c047cd9af56a66c2e00105 with peer 6d34e50fb5dd4a1db58013d4863b8992
I20240502 01:55:54.514559  6334 consensus_queue.cc:237] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:55:54.524497 10083 tablet_bootstrap.cc:492] T 02b106ec57b94e44a29947a80a030a23 P 60298ffea1dd4c25a38765c2ec9ec9d5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:54.525003 10083 tablet_bootstrap.cc:492] T 02b106ec57b94e44a29947a80a030a23 P 60298ffea1dd4c25a38765c2ec9ec9d5: Bootstrap complete.
I20240502 01:55:54.525332 10083 ts_tablet_manager.cc:1367] T 02b106ec57b94e44a29947a80a030a23 P 60298ffea1dd4c25a38765c2ec9ec9d5: Time spent bootstrapping tablet: real 0.221s	user 0.057s	sys 0.008s
I20240502 01:55:54.526281 10083 raft_consensus.cc:348] T 02b106ec57b94e44a29947a80a030a23 P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 01:55:54.526543 10083 raft_consensus.cc:729] T 02b106ec57b94e44a29947a80a030a23 P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60298ffea1dd4c25a38765c2ec9ec9d5, State: Initialized, Role: FOLLOWER
I20240502 01:55:54.526793 10083 consensus_queue.cc:260] T 02b106ec57b94e44a29947a80a030a23 P 60298ffea1dd4c25a38765c2ec9ec9d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 01:55:54.530565 10083 ts_tablet_manager.cc:1398] T 02b106ec57b94e44a29947a80a030a23 P 60298ffea1dd4c25a38765c2ec9ec9d5: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.001s
I20240502 01:55:54.536195  6489 tablet_copy_service.cc:342] P 07cdb213c77c4ffda1f999b66b725399: Request end of tablet copy session 60298ffea1dd4c25a38765c2ec9ec9d5-02b106ec57b94e44a29947a80a030a23 received from {username='slave'} at 127.25.109.133:42089
I20240502 01:55:54.536365  6489 tablet_copy_service.cc:433] P 07cdb213c77c4ffda1f999b66b725399: ending tablet copy session 60298ffea1dd4c25a38765c2ec9ec9d5-02b106ec57b94e44a29947a80a030a23 on tablet 02b106ec57b94e44a29947a80a030a23 with peer 60298ffea1dd4c25a38765c2ec9ec9d5
I20240502 01:55:54.547209  8270 raft_consensus.cc:1261] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:54.548283  9395 consensus_queue.cc:1035] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:54.557096  6467 raft_consensus.cc:1261] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:54.558066  9395 consensus_queue.cc:1035] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:54.564906  6066 raft_consensus.cc:1261] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:54.568445  9395 consensus_queue.cc:1035] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:54.575690  9705 raft_consensus.cc:2932] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:54.580454 10215 ts_tablet_manager.cc:902] T f309b40931dd444091869119eab33bbc P 9db17f84d37944098cbeb9e0d8456efb: Initiating tablet copy from peer 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867)
I20240502 01:55:54.592200 10352 ts_tablet_manager.cc:902] T 5be45a68cff24a75a03b2fcd4b7851f9 P 436822a397664c54b8d1b9eb3bc6cff4: Initiating tablet copy from peer 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227)
I20240502 01:55:54.592139  5922 catalog_manager.cc:5484] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 reported cstate change: config changed from index 4 to 5, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:54.595153  6467 raft_consensus.cc:2932] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:54.609146  6068 raft_consensus.cc:2932] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:54.619028  8405 raft_consensus.cc:1203] T c58d6b2401c047cd9af56a66c2e00105 P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 01:55:54.618584  8270 raft_consensus.cc:2932] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
W20240502 01:55:54.637997  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: afa5561f8c0e4c0fa9bbbd5da61348c7. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:54.643783 10215 tablet_copy_client.cc:308] T f309b40931dd444091869119eab33bbc P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.132:36867
I20240502 01:55:54.644829  6489 tablet_copy_service.cc:140] P 07cdb213c77c4ffda1f999b66b725399: Received BeginTabletCopySession request for tablet f309b40931dd444091869119eab33bbc from peer 9db17f84d37944098cbeb9e0d8456efb ({username='slave'} at 127.25.109.134:56199)
I20240502 01:55:54.645020  6489 tablet_copy_service.cc:161] P 07cdb213c77c4ffda1f999b66b725399: Beginning new tablet copy session on tablet f309b40931dd444091869119eab33bbc from peer 9db17f84d37944098cbeb9e0d8456efb at {username='slave'} at 127.25.109.134:56199: session id = 9db17f84d37944098cbeb9e0d8456efb-f309b40931dd444091869119eab33bbc
I20240502 01:55:54.648277  6489 tablet_copy_source_session.cc:205] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:54.650036 10215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f309b40931dd444091869119eab33bbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:54.658001 10255 tablet_copy_client.cc:308] T ca458907378e42d48adaa0ee647db4cd P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.130:38197
I20240502 01:55:54.659134  6224 tablet_copy_service.cc:140] P a2c90ee440b44b708bccc2e2d5385cc0: Received BeginTabletCopySession request for tablet ca458907378e42d48adaa0ee647db4cd from peer 6d34e50fb5dd4a1db58013d4863b8992 ({username='slave'} at 127.25.109.136:34777)
I20240502 01:55:54.659343  6224 tablet_copy_service.cc:161] P a2c90ee440b44b708bccc2e2d5385cc0: Beginning new tablet copy session on tablet ca458907378e42d48adaa0ee647db4cd from peer 6d34e50fb5dd4a1db58013d4863b8992 at {username='slave'} at 127.25.109.136:34777: session id = 6d34e50fb5dd4a1db58013d4863b8992-ca458907378e42d48adaa0ee647db4cd
I20240502 01:55:54.662366 10215 tablet_copy_client.cc:783] T f309b40931dd444091869119eab33bbc P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:54.662783  6224 tablet_copy_source_session.cc:205] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:54.663004 10215 tablet_copy_client.cc:647] T f309b40931dd444091869119eab33bbc P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:54.666396 10255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca458907378e42d48adaa0ee647db4cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:54.666739 10215 tablet_copy_client.cc:523] T f309b40931dd444091869119eab33bbc P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:54.667811 10352 tablet_copy_client.cc:308] T 5be45a68cff24a75a03b2fcd4b7851f9 P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.129:36227
I20240502 01:55:54.676187  6089 tablet_copy_service.cc:140] P 218bc8c1f9ce46059898fc738179adf9: Received BeginTabletCopySession request for tablet 5be45a68cff24a75a03b2fcd4b7851f9 from peer 436822a397664c54b8d1b9eb3bc6cff4 ({username='slave'} at 127.25.109.135:47139)
I20240502 01:55:54.676403  6089 tablet_copy_service.cc:161] P 218bc8c1f9ce46059898fc738179adf9: Beginning new tablet copy session on tablet 5be45a68cff24a75a03b2fcd4b7851f9 from peer 436822a397664c54b8d1b9eb3bc6cff4 at {username='slave'} at 127.25.109.135:47139: session id = 436822a397664c54b8d1b9eb3bc6cff4-5be45a68cff24a75a03b2fcd4b7851f9
I20240502 01:55:54.679459  6089 tablet_copy_source_session.cc:205] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:54.681529 10352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5be45a68cff24a75a03b2fcd4b7851f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:54.688609 10215 tablet_bootstrap.cc:492] T f309b40931dd444091869119eab33bbc P 9db17f84d37944098cbeb9e0d8456efb: Bootstrap starting.
I20240502 01:55:54.689251 10255 tablet_copy_client.cc:783] T ca458907378e42d48adaa0ee647db4cd P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:54.689584 10255 tablet_copy_client.cc:647] T ca458907378e42d48adaa0ee647db4cd P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:54.698211 10255 tablet_copy_client.cc:523] T ca458907378e42d48adaa0ee647db4cd P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:54.709177 10352 tablet_copy_client.cc:783] T 5be45a68cff24a75a03b2fcd4b7851f9 P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:54.709676 10352 tablet_copy_client.cc:647] T 5be45a68cff24a75a03b2fcd4b7851f9 P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:54.711755 10352 tablet_copy_client.cc:523] T 5be45a68cff24a75a03b2fcd4b7851f9 P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:54.716756 10352 tablet_bootstrap.cc:492] T 5be45a68cff24a75a03b2fcd4b7851f9 P 436822a397664c54b8d1b9eb3bc6cff4: Bootstrap starting.
I20240502 01:55:54.724419  8001 raft_consensus.cc:1203] T 02b106ec57b94e44a29947a80a030a23 P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:55:54.734095 10255 tablet_bootstrap.cc:492] T ca458907378e42d48adaa0ee647db4cd P 6d34e50fb5dd4a1db58013d4863b8992: Bootstrap starting.
I20240502 01:55:54.750169  6201 consensus_queue.cc:237] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:55:54.751749 10394 ts_tablet_manager.cc:902] T 2cefdfa8851147999c1425c606f36b1f P 436822a397664c54b8d1b9eb3bc6cff4: Initiating tablet copy from peer 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245)
W20240502 01:55:54.753645  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 086509f1ee0a4a81b2f8194b50dbe159. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:54.753867  8135 raft_consensus.cc:2932] T 5be45a68cff24a75a03b2fcd4b7851f9 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:55:54.756985  6068 raft_consensus.cc:1261] T 086509f1ee0a4a81b2f8194b50dbe159 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Refusing update from remote peer a2c90ee440b44b708bccc2e2d5385cc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:54.756431  8270 raft_consensus.cc:1261] T 086509f1ee0a4a81b2f8194b50dbe159 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Refusing update from remote peer a2c90ee440b44b708bccc2e2d5385cc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:54.758001 10193 consensus_queue.cc:1035] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:54.758636  6467 raft_consensus.cc:1261] T 086509f1ee0a4a81b2f8194b50dbe159 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Refusing update from remote peer a2c90ee440b44b708bccc2e2d5385cc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:54.762187 10197 ts_tablet_manager.cc:902] T 2cefdfa8851147999c1425c606f36b1f P 6d34e50fb5dd4a1db58013d4863b8992: Initiating tablet copy from peer 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245)
I20240502 01:55:54.763175  9945 consensus_queue.cc:1035] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:54.763989  9945 consensus_queue.cc:1035] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:54.785271 10398 raft_consensus.cc:2932] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:54.788367  8270 raft_consensus.cc:2932] T 086509f1ee0a4a81b2f8194b50dbe159 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:54.792537  6068 raft_consensus.cc:2932] T 086509f1ee0a4a81b2f8194b50dbe159 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:54.795254  5922 catalog_manager.cc:5484] T 086509f1ee0a4a81b2f8194b50dbe159 P 436822a397664c54b8d1b9eb3bc6cff4 reported cstate change: config changed from index 4 to 5, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New cstate: current_term: 1 leader_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:54.808451  6467 raft_consensus.cc:2932] T 086509f1ee0a4a81b2f8194b50dbe159 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:54.827059 10197 tablet_copy_client.cc:308] T 2cefdfa8851147999c1425c606f36b1f P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.131:44245
I20240502 01:55:54.830196  6357 tablet_copy_service.cc:140] P 94ffd7aaf85b4e17881241b8396fa2f3: Received BeginTabletCopySession request for tablet 2cefdfa8851147999c1425c606f36b1f from peer 6d34e50fb5dd4a1db58013d4863b8992 ({username='slave'} at 127.25.109.136:52593)
I20240502 01:55:54.830395  6357 tablet_copy_service.cc:161] P 94ffd7aaf85b4e17881241b8396fa2f3: Beginning new tablet copy session on tablet 2cefdfa8851147999c1425c606f36b1f from peer 6d34e50fb5dd4a1db58013d4863b8992 at {username='slave'} at 127.25.109.136:52593: session id = 6d34e50fb5dd4a1db58013d4863b8992-2cefdfa8851147999c1425c606f36b1f
I20240502 01:55:54.833379  6357 tablet_copy_source_session.cc:205] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:54.837761 10197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cefdfa8851147999c1425c606f36b1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:54.855937 10394 tablet_copy_client.cc:308] T 2cefdfa8851147999c1425c606f36b1f P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.131:44245
I20240502 01:55:54.856750  6357 tablet_copy_service.cc:140] P 94ffd7aaf85b4e17881241b8396fa2f3: Received BeginTabletCopySession request for tablet 2cefdfa8851147999c1425c606f36b1f from peer 436822a397664c54b8d1b9eb3bc6cff4 ({username='slave'} at 127.25.109.135:37865)
I20240502 01:55:54.856954  6357 tablet_copy_service.cc:161] P 94ffd7aaf85b4e17881241b8396fa2f3: Beginning new tablet copy session on tablet 2cefdfa8851147999c1425c606f36b1f from peer 436822a397664c54b8d1b9eb3bc6cff4 at {username='slave'} at 127.25.109.135:37865: session id = 436822a397664c54b8d1b9eb3bc6cff4-2cefdfa8851147999c1425c606f36b1f
I20240502 01:55:54.860111  6357 tablet_copy_source_session.cc:205] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:54.861778 10394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cefdfa8851147999c1425c606f36b1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:54.876130 10215 tablet_bootstrap.cc:492] T f309b40931dd444091869119eab33bbc P 9db17f84d37944098cbeb9e0d8456efb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:54.876724 10215 tablet_bootstrap.cc:492] T f309b40931dd444091869119eab33bbc P 9db17f84d37944098cbeb9e0d8456efb: Bootstrap complete.
I20240502 01:55:54.877105 10215 ts_tablet_manager.cc:1367] T f309b40931dd444091869119eab33bbc P 9db17f84d37944098cbeb9e0d8456efb: Time spent bootstrapping tablet: real 0.189s	user 0.042s	sys 0.010s
I20240502 01:55:54.878026 10215 raft_consensus.cc:348] T f309b40931dd444091869119eab33bbc P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:55:54.878424 10215 raft_consensus.cc:729] T f309b40931dd444091869119eab33bbc P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9db17f84d37944098cbeb9e0d8456efb, State: Initialized, Role: FOLLOWER
I20240502 01:55:54.878775 10215 consensus_queue.cc:260] T f309b40931dd444091869119eab33bbc P 9db17f84d37944098cbeb9e0d8456efb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:55:54.879555 10215 ts_tablet_manager.cc:1398] T f309b40931dd444091869119eab33bbc P 9db17f84d37944098cbeb9e0d8456efb: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:55:54.883204  6489 tablet_copy_service.cc:342] P 07cdb213c77c4ffda1f999b66b725399: Request end of tablet copy session 9db17f84d37944098cbeb9e0d8456efb-f309b40931dd444091869119eab33bbc received from {username='slave'} at 127.25.109.134:56199
I20240502 01:55:54.883384  6489 tablet_copy_service.cc:433] P 07cdb213c77c4ffda1f999b66b725399: ending tablet copy session 9db17f84d37944098cbeb9e0d8456efb-f309b40931dd444091869119eab33bbc on tablet f309b40931dd444091869119eab33bbc with peer 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:55:54.885557 10394 tablet_copy_client.cc:783] T 2cefdfa8851147999c1425c606f36b1f P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:54.885907 10394 tablet_copy_client.cc:647] T 2cefdfa8851147999c1425c606f36b1f P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:54.891314 10394 tablet_copy_client.cc:523] T 2cefdfa8851147999c1425c606f36b1f P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:54.897599 10394 tablet_bootstrap.cc:492] T 2cefdfa8851147999c1425c606f36b1f P 436822a397664c54b8d1b9eb3bc6cff4: Bootstrap starting.
I20240502 01:55:54.899811 10197 tablet_copy_client.cc:783] T 2cefdfa8851147999c1425c606f36b1f P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:54.900136 10197 tablet_copy_client.cc:647] T 2cefdfa8851147999c1425c606f36b1f P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:54.903582 10197 tablet_copy_client.cc:523] T 2cefdfa8851147999c1425c606f36b1f P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:54.907318 10197 tablet_bootstrap.cc:492] T 2cefdfa8851147999c1425c606f36b1f P 6d34e50fb5dd4a1db58013d4863b8992: Bootstrap starting.
I20240502 01:55:54.935292  6467 consensus_queue.cc:237] T 0039dd405cfc4dcea5e7353169e3cd52 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 01:55:54.946091  6334 raft_consensus.cc:1261] T 0039dd405cfc4dcea5e7353169e3cd52 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:55:54.947650  6402 consensus_peers.cc:485] T 0039dd405cfc4dcea5e7353169e3cd52 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133:43379): Couldn't send request to peer 60298ffea1dd4c25a38765c2ec9ec9d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0039dd405cfc4dcea5e7353169e3cd52. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:54.948768 10064 consensus_queue.cc:1035] T 0039dd405cfc4dcea5e7353169e3cd52 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:54.972039  6204 raft_consensus.cc:1261] T 0039dd405cfc4dcea5e7353169e3cd52 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:54.974062 10450 consensus_queue.cc:1035] T 0039dd405cfc4dcea5e7353169e3cd52 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:54.983225 10255 tablet_bootstrap.cc:492] T ca458907378e42d48adaa0ee647db4cd P 6d34e50fb5dd4a1db58013d4863b8992: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:54.983743 10255 tablet_bootstrap.cc:492] T ca458907378e42d48adaa0ee647db4cd P 6d34e50fb5dd4a1db58013d4863b8992: Bootstrap complete.
I20240502 01:55:54.984125 10255 ts_tablet_manager.cc:1367] T ca458907378e42d48adaa0ee647db4cd P 6d34e50fb5dd4a1db58013d4863b8992: Time spent bootstrapping tablet: real 0.250s	user 0.057s	sys 0.023s
I20240502 01:55:54.985029 10255 raft_consensus.cc:348] T ca458907378e42d48adaa0ee647db4cd P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:55:54.985317 10255 raft_consensus.cc:729] T ca458907378e42d48adaa0ee647db4cd P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d34e50fb5dd4a1db58013d4863b8992, State: Initialized, Role: FOLLOWER
I20240502 01:55:54.985548 10352 tablet_bootstrap.cc:492] T 5be45a68cff24a75a03b2fcd4b7851f9 P 436822a397664c54b8d1b9eb3bc6cff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:54.985713 10394 tablet_bootstrap.cc:492] T 2cefdfa8851147999c1425c606f36b1f P 436822a397664c54b8d1b9eb3bc6cff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:54.990674 10255 consensus_queue.cc:260] T ca458907378e42d48adaa0ee647db4cd P 6d34e50fb5dd4a1db58013d4863b8992 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:55:54.992995 10255 ts_tablet_manager.cc:1398] T ca458907378e42d48adaa0ee647db4cd P 6d34e50fb5dd4a1db58013d4863b8992: Time spent starting tablet: real 0.009s	user 0.002s	sys 0.000s
I20240502 01:55:54.995150  6224 tablet_copy_service.cc:342] P a2c90ee440b44b708bccc2e2d5385cc0: Request end of tablet copy session 6d34e50fb5dd4a1db58013d4863b8992-ca458907378e42d48adaa0ee647db4cd received from {username='slave'} at 127.25.109.136:34777
I20240502 01:55:54.995335  6224 tablet_copy_service.cc:433] P a2c90ee440b44b708bccc2e2d5385cc0: ending tablet copy session 6d34e50fb5dd4a1db58013d4863b8992-ca458907378e42d48adaa0ee647db4cd on tablet ca458907378e42d48adaa0ee647db4cd with peer 6d34e50fb5dd4a1db58013d4863b8992
I20240502 01:55:55.000433 10352 tablet_bootstrap.cc:492] T 5be45a68cff24a75a03b2fcd4b7851f9 P 436822a397664c54b8d1b9eb3bc6cff4: Bootstrap complete.
I20240502 01:55:55.000799 10352 ts_tablet_manager.cc:1367] T 5be45a68cff24a75a03b2fcd4b7851f9 P 436822a397664c54b8d1b9eb3bc6cff4: Time spent bootstrapping tablet: real 0.284s	user 0.048s	sys 0.004s
I20240502 01:55:55.001690 10352 raft_consensus.cc:348] T 5be45a68cff24a75a03b2fcd4b7851f9 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:55:55.001989 10352 raft_consensus.cc:729] T 5be45a68cff24a75a03b2fcd4b7851f9 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 436822a397664c54b8d1b9eb3bc6cff4, State: Initialized, Role: FOLLOWER
I20240502 01:55:55.002257 10352 consensus_queue.cc:260] T 5be45a68cff24a75a03b2fcd4b7851f9 P 436822a397664c54b8d1b9eb3bc6cff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:55:55.002842 10352 ts_tablet_manager.cc:1398] T 5be45a68cff24a75a03b2fcd4b7851f9 P 436822a397664c54b8d1b9eb3bc6cff4: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 01:55:55.003698  6089 tablet_copy_service.cc:342] P 218bc8c1f9ce46059898fc738179adf9: Request end of tablet copy session 436822a397664c54b8d1b9eb3bc6cff4-5be45a68cff24a75a03b2fcd4b7851f9 received from {username='slave'} at 127.25.109.135:47139
I20240502 01:55:55.003878  6089 tablet_copy_service.cc:433] P 218bc8c1f9ce46059898fc738179adf9: ending tablet copy session 436822a397664c54b8d1b9eb3bc6cff4-5be45a68cff24a75a03b2fcd4b7851f9 on tablet 5be45a68cff24a75a03b2fcd4b7851f9 with peer 436822a397664c54b8d1b9eb3bc6cff4
I20240502 01:55:55.004161 10394 tablet_bootstrap.cc:492] T 2cefdfa8851147999c1425c606f36b1f P 436822a397664c54b8d1b9eb3bc6cff4: Bootstrap complete.
I20240502 01:55:55.004482 10394 ts_tablet_manager.cc:1367] T 2cefdfa8851147999c1425c606f36b1f P 436822a397664c54b8d1b9eb3bc6cff4: Time spent bootstrapping tablet: real 0.107s	user 0.054s	sys 0.000s
I20240502 01:55:55.005336 10394 raft_consensus.cc:348] T 2cefdfa8851147999c1425c606f36b1f P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:55:55.005597 10394 raft_consensus.cc:729] T 2cefdfa8851147999c1425c606f36b1f P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 436822a397664c54b8d1b9eb3bc6cff4, State: Initialized, Role: FOLLOWER
I20240502 01:55:55.005841 10394 consensus_queue.cc:260] T 2cefdfa8851147999c1425c606f36b1f P 436822a397664c54b8d1b9eb3bc6cff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:55:55.006515 10394 ts_tablet_manager.cc:1398] T 2cefdfa8851147999c1425c606f36b1f P 436822a397664c54b8d1b9eb3bc6cff4: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:55:55.007838  6357 tablet_copy_service.cc:342] P 94ffd7aaf85b4e17881241b8396fa2f3: Request end of tablet copy session 436822a397664c54b8d1b9eb3bc6cff4-2cefdfa8851147999c1425c606f36b1f received from {username='slave'} at 127.25.109.135:37865
I20240502 01:55:55.008004  6357 tablet_copy_service.cc:433] P 94ffd7aaf85b4e17881241b8396fa2f3: ending tablet copy session 436822a397664c54b8d1b9eb3bc6cff4-2cefdfa8851147999c1425c606f36b1f on tablet 2cefdfa8851147999c1425c606f36b1f with peer 436822a397664c54b8d1b9eb3bc6cff4
I20240502 01:55:55.026515 10215 ts_tablet_manager.cc:902] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 9db17f84d37944098cbeb9e0d8456efb: Initiating tablet copy from peer 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245)
I20240502 01:55:55.042651 10197 tablet_bootstrap.cc:492] T 2cefdfa8851147999c1425c606f36b1f P 6d34e50fb5dd4a1db58013d4863b8992: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:55.043145 10197 tablet_bootstrap.cc:492] T 2cefdfa8851147999c1425c606f36b1f P 6d34e50fb5dd4a1db58013d4863b8992: Bootstrap complete.
I20240502 01:55:55.043447 10197 ts_tablet_manager.cc:1367] T 2cefdfa8851147999c1425c606f36b1f P 6d34e50fb5dd4a1db58013d4863b8992: Time spent bootstrapping tablet: real 0.136s	user 0.059s	sys 0.018s
W20240502 01:55:55.048485  6403 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.130:38197, user_credentials={real_user=slave}} blocked reactor thread for 61626.6us
I20240502 01:55:55.050249 10450 raft_consensus.cc:2932] T 0039dd405cfc4dcea5e7353169e3cd52 P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:55:55.055976  6335 raft_consensus.cc:2932] T 0039dd405cfc4dcea5e7353169e3cd52 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:55:55.061211  8405 raft_consensus.cc:1203] T ca458907378e42d48adaa0ee647db4cd P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 01:55:55.063345  6204 raft_consensus.cc:2932] T 0039dd405cfc4dcea5e7353169e3cd52 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:55:55.066627 10197 raft_consensus.cc:348] T 2cefdfa8851147999c1425c606f36b1f P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:55:55.068903  5922 catalog_manager.cc:5484] T 0039dd405cfc4dcea5e7353169e3cd52 P 94ffd7aaf85b4e17881241b8396fa2f3 reported cstate change: config changed from index -1 to 4, VOTER 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133) added. New cstate: current_term: 1 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:55:55.066969 10197 raft_consensus.cc:729] T 2cefdfa8851147999c1425c606f36b1f P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d34e50fb5dd4a1db58013d4863b8992, State: Initialized, Role: FOLLOWER
I20240502 01:55:55.082437 10197 consensus_queue.cc:260] T 2cefdfa8851147999c1425c606f36b1f P 6d34e50fb5dd4a1db58013d4863b8992 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:55:55.083231 10197 ts_tablet_manager.cc:1398] T 2cefdfa8851147999c1425c606f36b1f P 6d34e50fb5dd4a1db58013d4863b8992: Time spent starting tablet: real 0.040s	user 0.005s	sys 0.000s
I20240502 01:55:55.088080 10215 tablet_copy_client.cc:308] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.131:44245
I20240502 01:55:55.095422  6467 consensus_queue.cc:237] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:55:55.104481  6357 tablet_copy_service.cc:342] P 94ffd7aaf85b4e17881241b8396fa2f3: Request end of tablet copy session 6d34e50fb5dd4a1db58013d4863b8992-2cefdfa8851147999c1425c606f36b1f received from {username='slave'} at 127.25.109.136:52593
I20240502 01:55:55.104635  6357 tablet_copy_service.cc:433] P 94ffd7aaf85b4e17881241b8396fa2f3: ending tablet copy session 6d34e50fb5dd4a1db58013d4863b8992-2cefdfa8851147999c1425c606f36b1f on tablet 2cefdfa8851147999c1425c606f36b1f with peer 6d34e50fb5dd4a1db58013d4863b8992
I20240502 01:55:55.108374  8001 raft_consensus.cc:1261] T 6984d638991442e3adaed1b24bee95e9 P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:55.110157  8134 raft_consensus.cc:1203] T f309b40931dd444091869119eab33bbc P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
W20240502 01:55:55.112891  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6984d638991442e3adaed1b24bee95e9. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:55.113596 10469 consensus_queue.cc:1035] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:55.114202  6066 raft_consensus.cc:1261] T 6984d638991442e3adaed1b24bee95e9 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:55.119711  6357 tablet_copy_service.cc:140] P 94ffd7aaf85b4e17881241b8396fa2f3: Received BeginTabletCopySession request for tablet afa5561f8c0e4c0fa9bbbd5da61348c7 from peer 9db17f84d37944098cbeb9e0d8456efb ({username='slave'} at 127.25.109.134:41767)
I20240502 01:55:55.119932  6357 tablet_copy_service.cc:161] P 94ffd7aaf85b4e17881241b8396fa2f3: Beginning new tablet copy session on tablet afa5561f8c0e4c0fa9bbbd5da61348c7 from peer 9db17f84d37944098cbeb9e0d8456efb at {username='slave'} at 127.25.109.134:41767: session id = 9db17f84d37944098cbeb9e0d8456efb-afa5561f8c0e4c0fa9bbbd5da61348c7
I20240502 01:55:55.122112  8269 raft_consensus.cc:1203] T 5be45a68cff24a75a03b2fcd4b7851f9 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 01:55:55.122700  6357 tablet_copy_source_session.cc:205] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:55.124820  8405 raft_consensus.cc:1261] T 6984d638991442e3adaed1b24bee95e9 P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:55.124948 10215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afa5561f8c0e4c0fa9bbbd5da61348c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:55.126854 10064 consensus_queue.cc:1035] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:55.132501  6335 raft_consensus.cc:1261] T 6984d638991442e3adaed1b24bee95e9 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:55.132560 10469 consensus_queue.cc:1035] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:55.134817 10450 consensus_queue.cc:1035] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:55.137609 10215 tablet_copy_client.cc:783] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:55.137933 10215 tablet_copy_client.cc:647] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:55.140306 10215 tablet_copy_client.cc:523] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:55.153347 10215 tablet_bootstrap.cc:492] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 9db17f84d37944098cbeb9e0d8456efb: Bootstrap starting.
I20240502 01:55:55.184044 10480 raft_consensus.cc:2932] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:55.200039  6066 raft_consensus.cc:2932] T 6984d638991442e3adaed1b24bee95e9 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:55.212033  6335 raft_consensus.cc:2932] T 6984d638991442e3adaed1b24bee95e9 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:55.214820  8405 raft_consensus.cc:2932] T 6984d638991442e3adaed1b24bee95e9 P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:55.223963  8269 raft_consensus.cc:1203] T 2cefdfa8851147999c1425c606f36b1f P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 01:55:55.213924  5912 catalog_manager.cc:5484] T 6984d638991442e3adaed1b24bee95e9 P 218bc8c1f9ce46059898fc738179adf9 reported cstate change: config changed from index 5 to 6, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New cstate: current_term: 1 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:55.235338  8001 raft_consensus.cc:2932] T 6984d638991442e3adaed1b24bee95e9 P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:55.254690 10215 tablet_bootstrap.cc:492] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 9db17f84d37944098cbeb9e0d8456efb: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:55.255252 10215 tablet_bootstrap.cc:492] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 9db17f84d37944098cbeb9e0d8456efb: Bootstrap complete.
I20240502 01:55:55.255611 10215 ts_tablet_manager.cc:1367] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 9db17f84d37944098cbeb9e0d8456efb: Time spent bootstrapping tablet: real 0.102s	user 0.032s	sys 0.009s
I20240502 01:55:55.262765  8405 raft_consensus.cc:1203] T 2cefdfa8851147999c1425c606f36b1f P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 01:55:55.279135 10215 raft_consensus.cc:348] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:55:55.279500 10215 raft_consensus.cc:729] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9db17f84d37944098cbeb9e0d8456efb, State: Initialized, Role: FOLLOWER
I20240502 01:55:55.281291 10215 consensus_queue.cc:260] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 9db17f84d37944098cbeb9e0d8456efb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:55:55.282509 10215 ts_tablet_manager.cc:1398] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 9db17f84d37944098cbeb9e0d8456efb: Time spent starting tablet: real 0.027s	user 0.003s	sys 0.003s
I20240502 01:55:55.285003  6357 tablet_copy_service.cc:342] P 94ffd7aaf85b4e17881241b8396fa2f3: Request end of tablet copy session 9db17f84d37944098cbeb9e0d8456efb-afa5561f8c0e4c0fa9bbbd5da61348c7 received from {username='slave'} at 127.25.109.134:41767
I20240502 01:55:55.285167  6357 tablet_copy_service.cc:433] P 94ffd7aaf85b4e17881241b8396fa2f3: ending tablet copy session 9db17f84d37944098cbeb9e0d8456efb-afa5561f8c0e4c0fa9bbbd5da61348c7 on tablet afa5561f8c0e4c0fa9bbbd5da61348c7 with peer 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:55:55.372612 10197 ts_tablet_manager.cc:902] T 086509f1ee0a4a81b2f8194b50dbe159 P 6d34e50fb5dd4a1db58013d4863b8992: Initiating tablet copy from peer a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197)
I20240502 01:55:55.387352  6046 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:55:55.418881  8135 raft_consensus.cc:1203] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 01:55:55.428045  6447 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:55:55.460932  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:55.480218  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:55.511691 10552 ts_tablet_manager.cc:902] T 0039dd405cfc4dcea5e7353169e3cd52 P 60298ffea1dd4c25a38765c2ec9ec9d5: Initiating tablet copy from peer 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867)
I20240502 01:55:55.564924 10197 tablet_copy_client.cc:308] T 086509f1ee0a4a81b2f8194b50dbe159 P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.130:38197
I20240502 01:55:55.567070  6224 tablet_copy_service.cc:140] P a2c90ee440b44b708bccc2e2d5385cc0: Received BeginTabletCopySession request for tablet 086509f1ee0a4a81b2f8194b50dbe159 from peer 6d34e50fb5dd4a1db58013d4863b8992 ({username='slave'} at 127.25.109.136:34777)
I20240502 01:55:55.567309  6224 tablet_copy_service.cc:161] P a2c90ee440b44b708bccc2e2d5385cc0: Beginning new tablet copy session on tablet 086509f1ee0a4a81b2f8194b50dbe159 from peer 6d34e50fb5dd4a1db58013d4863b8992 at {username='slave'} at 127.25.109.136:34777: session id = 6d34e50fb5dd4a1db58013d4863b8992-086509f1ee0a4a81b2f8194b50dbe159
I20240502 01:55:55.571647  6224 tablet_copy_source_session.cc:205] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:55.574146 10215 ts_tablet_manager.cc:902] T 6984d638991442e3adaed1b24bee95e9 P 9db17f84d37944098cbeb9e0d8456efb: Initiating tablet copy from peer 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867)
I20240502 01:55:55.576177  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:55:55.590751 10552 tablet_copy_client.cc:308] T 0039dd405cfc4dcea5e7353169e3cd52 P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.132:36867
I20240502 01:55:55.592676  6489 tablet_copy_service.cc:140] P 07cdb213c77c4ffda1f999b66b725399: Received BeginTabletCopySession request for tablet 0039dd405cfc4dcea5e7353169e3cd52 from peer 60298ffea1dd4c25a38765c2ec9ec9d5 ({username='slave'} at 127.25.109.133:42089)
I20240502 01:55:55.592934  6489 tablet_copy_service.cc:161] P 07cdb213c77c4ffda1f999b66b725399: Beginning new tablet copy session on tablet 0039dd405cfc4dcea5e7353169e3cd52 from peer 60298ffea1dd4c25a38765c2ec9ec9d5 at {username='slave'} at 127.25.109.133:42089: session id = 60298ffea1dd4c25a38765c2ec9ec9d5-0039dd405cfc4dcea5e7353169e3cd52
I20240502 01:55:55.596441  6489 tablet_copy_source_session.cc:205] T 0039dd405cfc4dcea5e7353169e3cd52 P 07cdb213c77c4ffda1f999b66b725399: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:55.598675 10552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0039dd405cfc4dcea5e7353169e3cd52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:55.606446 10197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 086509f1ee0a4a81b2f8194b50dbe159. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:55.624861  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:55.625761 10552 tablet_copy_client.cc:783] T 0039dd405cfc4dcea5e7353169e3cd52 P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:55.627328 10552 tablet_copy_client.cc:647] T 0039dd405cfc4dcea5e7353169e3cd52 P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:55.629607 10552 tablet_copy_client.cc:523] T 0039dd405cfc4dcea5e7353169e3cd52 P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:55.641419 10552 tablet_bootstrap.cc:492] T 0039dd405cfc4dcea5e7353169e3cd52 P 60298ffea1dd4c25a38765c2ec9ec9d5: Bootstrap starting.
I20240502 01:55:55.646517 10215 tablet_copy_client.cc:308] T 6984d638991442e3adaed1b24bee95e9 P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.132:36867
I20240502 01:55:55.647305  6489 tablet_copy_service.cc:140] P 07cdb213c77c4ffda1f999b66b725399: Received BeginTabletCopySession request for tablet 6984d638991442e3adaed1b24bee95e9 from peer 9db17f84d37944098cbeb9e0d8456efb ({username='slave'} at 127.25.109.134:56199)
I20240502 01:55:55.647511  6489 tablet_copy_service.cc:161] P 07cdb213c77c4ffda1f999b66b725399: Beginning new tablet copy session on tablet 6984d638991442e3adaed1b24bee95e9 from peer 9db17f84d37944098cbeb9e0d8456efb at {username='slave'} at 127.25.109.134:56199: session id = 9db17f84d37944098cbeb9e0d8456efb-6984d638991442e3adaed1b24bee95e9
I20240502 01:55:55.650836  6489 tablet_copy_source_session.cc:205] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:55.652585 10215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6984d638991442e3adaed1b24bee95e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:55.674435 10215 tablet_copy_client.cc:783] T 6984d638991442e3adaed1b24bee95e9 P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:55.674738 10215 tablet_copy_client.cc:647] T 6984d638991442e3adaed1b24bee95e9 P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:55.685870 10215 tablet_copy_client.cc:523] T 6984d638991442e3adaed1b24bee95e9 P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:55.692814 10215 tablet_bootstrap.cc:492] T 6984d638991442e3adaed1b24bee95e9 P 9db17f84d37944098cbeb9e0d8456efb: Bootstrap starting.
I20240502 01:55:55.716629  8385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:55.716761 10552 tablet_bootstrap.cc:492] T 0039dd405cfc4dcea5e7353169e3cd52 P 60298ffea1dd4c25a38765c2ec9ec9d5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:55.717224 10552 tablet_bootstrap.cc:492] T 0039dd405cfc4dcea5e7353169e3cd52 P 60298ffea1dd4c25a38765c2ec9ec9d5: Bootstrap complete.
I20240502 01:55:55.717540 10552 ts_tablet_manager.cc:1367] T 0039dd405cfc4dcea5e7353169e3cd52 P 60298ffea1dd4c25a38765c2ec9ec9d5: Time spent bootstrapping tablet: real 0.076s	user 0.045s	sys 0.014s
I20240502 01:55:55.718473 10552 raft_consensus.cc:348] T 0039dd405cfc4dcea5e7353169e3cd52 P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 01:55:55.718729 10552 raft_consensus.cc:729] T 0039dd405cfc4dcea5e7353169e3cd52 P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60298ffea1dd4c25a38765c2ec9ec9d5, State: Initialized, Role: FOLLOWER
I20240502 01:55:55.719018 10552 consensus_queue.cc:260] T 0039dd405cfc4dcea5e7353169e3cd52 P 60298ffea1dd4c25a38765c2ec9ec9d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 01:55:55.719761 10552 ts_tablet_manager.cc:1398] T 0039dd405cfc4dcea5e7353169e3cd52 P 60298ffea1dd4c25a38765c2ec9ec9d5: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 01:55:55.720820  6489 tablet_copy_service.cc:342] P 07cdb213c77c4ffda1f999b66b725399: Request end of tablet copy session 60298ffea1dd4c25a38765c2ec9ec9d5-0039dd405cfc4dcea5e7353169e3cd52 received from {username='slave'} at 127.25.109.133:42089
I20240502 01:55:55.720965  6489 tablet_copy_service.cc:433] P 07cdb213c77c4ffda1f999b66b725399: ending tablet copy session 60298ffea1dd4c25a38765c2ec9ec9d5-0039dd405cfc4dcea5e7353169e3cd52 on tablet 0039dd405cfc4dcea5e7353169e3cd52 with peer 60298ffea1dd4c25a38765c2ec9ec9d5
I20240502 01:55:55.753980  6335 consensus_queue.cc:237] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:55:55.785266 10215 tablet_bootstrap.cc:492] T 6984d638991442e3adaed1b24bee95e9 P 9db17f84d37944098cbeb9e0d8456efb: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20240502 01:55:55.787815  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3a7698f6bd4349ae832a22c853d76fea. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:55.789317  8001 raft_consensus.cc:1261] T 3a7698f6bd4349ae832a22c853d76fea P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:55.789731  6066 raft_consensus.cc:1261] T 3a7698f6bd4349ae832a22c853d76fea P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:55.790707 10578 consensus_queue.cc:1035] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:55.791357 10578 consensus_queue.cc:1035] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:55.802214 10215 tablet_bootstrap.cc:492] T 6984d638991442e3adaed1b24bee95e9 P 9db17f84d37944098cbeb9e0d8456efb: Bootstrap complete.
I20240502 01:55:55.802603 10215 ts_tablet_manager.cc:1367] T 6984d638991442e3adaed1b24bee95e9 P 9db17f84d37944098cbeb9e0d8456efb: Time spent bootstrapping tablet: real 0.110s	user 0.039s	sys 0.009s
I20240502 01:55:55.803500 10215 raft_consensus.cc:348] T 6984d638991442e3adaed1b24bee95e9 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:55:55.805148  6204 raft_consensus.cc:1261] T 3a7698f6bd4349ae832a22c853d76fea P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:55.806090  9705 consensus_queue.cc:1035] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:55.806489 10197 tablet_copy_client.cc:783] T 086509f1ee0a4a81b2f8194b50dbe159 P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:55.806761 10197 tablet_copy_client.cc:647] T 086509f1ee0a4a81b2f8194b50dbe159 P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:55.810778  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:55.815339 10215 raft_consensus.cc:729] T 6984d638991442e3adaed1b24bee95e9 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9db17f84d37944098cbeb9e0d8456efb, State: Initialized, Role: FOLLOWER
I20240502 01:55:55.818709 10215 consensus_queue.cc:260] T 6984d638991442e3adaed1b24bee95e9 P 9db17f84d37944098cbeb9e0d8456efb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:55:55.822098 10215 ts_tablet_manager.cc:1398] T 6984d638991442e3adaed1b24bee95e9 P 9db17f84d37944098cbeb9e0d8456efb: Time spent starting tablet: real 0.019s	user 0.003s	sys 0.000s
I20240502 01:55:55.824190  6489 tablet_copy_service.cc:342] P 07cdb213c77c4ffda1f999b66b725399: Request end of tablet copy session 9db17f84d37944098cbeb9e0d8456efb-6984d638991442e3adaed1b24bee95e9 received from {username='slave'} at 127.25.109.134:56199
I20240502 01:55:55.824370  6489 tablet_copy_service.cc:433] P 07cdb213c77c4ffda1f999b66b725399: ending tablet copy session 9db17f84d37944098cbeb9e0d8456efb-6984d638991442e3adaed1b24bee95e9 on tablet 6984d638991442e3adaed1b24bee95e9 with peer 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:55:55.847800  6447 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:55:55.851370 10197 tablet_copy_client.cc:523] T 086509f1ee0a4a81b2f8194b50dbe159 P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:55.853096 10578 raft_consensus.cc:2932] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:55.861985  8001 raft_consensus.cc:2932] T 3a7698f6bd4349ae832a22c853d76fea P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:55.871479  6066 raft_consensus.cc:2932] T 3a7698f6bd4349ae832a22c853d76fea P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:55.881186  5921 catalog_manager.cc:5484] T 3a7698f6bd4349ae832a22c853d76fea P 60298ffea1dd4c25a38765c2ec9ec9d5 reported cstate change: config changed from index 4 to 5, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:55.892752  6046 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:55:55.924002  6201 raft_consensus.cc:2932] T 3a7698f6bd4349ae832a22c853d76fea P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:55.954180 10197 tablet_bootstrap.cc:492] T 086509f1ee0a4a81b2f8194b50dbe159 P 6d34e50fb5dd4a1db58013d4863b8992: Bootstrap starting.
I20240502 01:55:55.980454  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:55.980954  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:56.020697  8385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:56.080590  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:56.089077  8001 raft_consensus.cc:1203] T 0039dd405cfc4dcea5e7353169e3cd52 P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:55:56.145143  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:55:56.149902  8135 raft_consensus.cc:1203] T 6984d638991442e3adaed1b24bee95e9 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6]   Dedup: 1.6->[]
I20240502 01:55:56.201439  6046 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:55:56.213469  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:56.265058  6335 consensus_queue.cc:237] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:55:56.268152  8385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:56.270844  6447 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 01:55:56.282068  6268 consensus_peers.cc:485] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135:40543): Couldn't send request to peer 436822a397664c54b8d1b9eb3bc6cff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 866f87ea9f7f48edb667b1b2eb3fa239. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:56.283313  6467 raft_consensus.cc:1261] T 866f87ea9f7f48edb667b1b2eb3fa239 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:56.284070 10610 consensus_queue.cc:1035] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:56.287608  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:56.289319  6201 raft_consensus.cc:1261] T 866f87ea9f7f48edb667b1b2eb3fa239 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:56.289924 10610 consensus_queue.cc:1035] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:56.300145  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:55:56.332758  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:56.334429  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:56.355357  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:56.371124 10610 raft_consensus.cc:2932] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:55:56.373041  6467 raft_consensus.cc:2932] T 866f87ea9f7f48edb667b1b2eb3fa239 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:55:56.384627  5912 catalog_manager.cc:5484] T 866f87ea9f7f48edb667b1b2eb3fa239 P 07cdb213c77c4ffda1f999b66b725399 reported cstate change: config changed from index -1 to 4, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:55:56.384673  6201 raft_consensus.cc:2932] T 866f87ea9f7f48edb667b1b2eb3fa239 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:55:56.396943 10668 ts_tablet_manager.cc:902] T 3a7698f6bd4349ae832a22c853d76fea P 6d34e50fb5dd4a1db58013d4863b8992: Initiating tablet copy from peer 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245)
I20240502 01:55:56.410923  8385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:56.419036  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:56.441211  6046 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:55:56.446888  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:56.484238  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:56.510188 10668 tablet_copy_client.cc:308] T 3a7698f6bd4349ae832a22c853d76fea P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.131:44245
I20240502 01:55:56.511087  6357 tablet_copy_service.cc:140] P 94ffd7aaf85b4e17881241b8396fa2f3: Received BeginTabletCopySession request for tablet 3a7698f6bd4349ae832a22c853d76fea from peer 6d34e50fb5dd4a1db58013d4863b8992 ({username='slave'} at 127.25.109.136:52593)
I20240502 01:55:56.511269  6357 tablet_copy_service.cc:161] P 94ffd7aaf85b4e17881241b8396fa2f3: Beginning new tablet copy session on tablet 3a7698f6bd4349ae832a22c853d76fea from peer 6d34e50fb5dd4a1db58013d4863b8992 at {username='slave'} at 127.25.109.136:52593: session id = 6d34e50fb5dd4a1db58013d4863b8992-3a7698f6bd4349ae832a22c853d76fea
I20240502 01:55:56.514320  6357 tablet_copy_source_session.cc:205] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:56.516816 10668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a7698f6bd4349ae832a22c853d76fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:56.560662  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:55:56.561580 10668 tablet_copy_client.cc:783] T 3a7698f6bd4349ae832a22c853d76fea P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:56.561749  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:56.561939 10668 tablet_copy_client.cc:647] T 3a7698f6bd4349ae832a22c853d76fea P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:56.564793 10668 tablet_copy_client.cc:523] T 3a7698f6bd4349ae832a22c853d76fea P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:56.571605 10668 tablet_bootstrap.cc:492] T 3a7698f6bd4349ae832a22c853d76fea P 6d34e50fb5dd4a1db58013d4863b8992: Bootstrap starting.
I20240502 01:55:56.595306  6447 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:55:56.661613 10197 tablet_bootstrap.cc:492] T 086509f1ee0a4a81b2f8194b50dbe159 P 6d34e50fb5dd4a1db58013d4863b8992: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:56.662190 10197 tablet_bootstrap.cc:492] T 086509f1ee0a4a81b2f8194b50dbe159 P 6d34e50fb5dd4a1db58013d4863b8992: Bootstrap complete.
I20240502 01:55:56.662529 10197 ts_tablet_manager.cc:1367] T 086509f1ee0a4a81b2f8194b50dbe159 P 6d34e50fb5dd4a1db58013d4863b8992: Time spent bootstrapping tablet: real 0.708s	user 0.041s	sys 0.013s
I20240502 01:55:56.663486 10197 raft_consensus.cc:348] T 086509f1ee0a4a81b2f8194b50dbe159 P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:55:56.663758 10197 raft_consensus.cc:729] T 086509f1ee0a4a81b2f8194b50dbe159 P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d34e50fb5dd4a1db58013d4863b8992, State: Initialized, Role: FOLLOWER
I20240502 01:55:56.680961 10197 consensus_queue.cc:260] T 086509f1ee0a4a81b2f8194b50dbe159 P 6d34e50fb5dd4a1db58013d4863b8992 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:55:56.729101 10197 ts_tablet_manager.cc:1398] T 086509f1ee0a4a81b2f8194b50dbe159 P 6d34e50fb5dd4a1db58013d4863b8992: Time spent starting tablet: real 0.066s	user 0.012s	sys 0.002s
I20240502 01:55:56.734042  6224 tablet_copy_service.cc:342] P a2c90ee440b44b708bccc2e2d5385cc0: Request end of tablet copy session 6d34e50fb5dd4a1db58013d4863b8992-086509f1ee0a4a81b2f8194b50dbe159 received from {username='slave'} at 127.25.109.136:34777
I20240502 01:55:56.734192  6224 tablet_copy_service.cc:433] P a2c90ee440b44b708bccc2e2d5385cc0: ending tablet copy session 6d34e50fb5dd4a1db58013d4863b8992-086509f1ee0a4a81b2f8194b50dbe159 on tablet 086509f1ee0a4a81b2f8194b50dbe159 with peer 6d34e50fb5dd4a1db58013d4863b8992
I20240502 01:55:56.826505 10668 tablet_bootstrap.cc:492] T 3a7698f6bd4349ae832a22c853d76fea P 6d34e50fb5dd4a1db58013d4863b8992: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:56.827674 10668 tablet_bootstrap.cc:492] T 3a7698f6bd4349ae832a22c853d76fea P 6d34e50fb5dd4a1db58013d4863b8992: Bootstrap complete.
I20240502 01:55:56.828123 10668 ts_tablet_manager.cc:1367] T 3a7698f6bd4349ae832a22c853d76fea P 6d34e50fb5dd4a1db58013d4863b8992: Time spent bootstrapping tablet: real 0.257s	user 0.062s	sys 0.021s
I20240502 01:55:56.829113 10668 raft_consensus.cc:348] T 3a7698f6bd4349ae832a22c853d76fea P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:55:56.830653 10668 raft_consensus.cc:729] T 3a7698f6bd4349ae832a22c853d76fea P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d34e50fb5dd4a1db58013d4863b8992, State: Initialized, Role: FOLLOWER
I20240502 01:55:56.830993 10668 consensus_queue.cc:260] T 3a7698f6bd4349ae832a22c853d76fea P 6d34e50fb5dd4a1db58013d4863b8992 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:55:56.833065 10668 ts_tablet_manager.cc:1398] T 3a7698f6bd4349ae832a22c853d76fea P 6d34e50fb5dd4a1db58013d4863b8992: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240502 01:55:56.833846  6357 tablet_copy_service.cc:342] P 94ffd7aaf85b4e17881241b8396fa2f3: Request end of tablet copy session 6d34e50fb5dd4a1db58013d4863b8992-3a7698f6bd4349ae832a22c853d76fea received from {username='slave'} at 127.25.109.136:52593
I20240502 01:55:56.833985  6357 tablet_copy_service.cc:433] P 94ffd7aaf85b4e17881241b8396fa2f3: ending tablet copy session 6d34e50fb5dd4a1db58013d4863b8992-3a7698f6bd4349ae832a22c853d76fea on tablet 3a7698f6bd4349ae832a22c853d76fea with peer 6d34e50fb5dd4a1db58013d4863b8992
I20240502 01:55:56.869367 10719 ts_tablet_manager.cc:902] T 866f87ea9f7f48edb667b1b2eb3fa239 P 436822a397664c54b8d1b9eb3bc6cff4: Initiating tablet copy from peer 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245)
I20240502 01:55:56.909756 10719 tablet_copy_client.cc:308] T 866f87ea9f7f48edb667b1b2eb3fa239 P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.131:44245
I20240502 01:55:56.910995  6357 tablet_copy_service.cc:140] P 94ffd7aaf85b4e17881241b8396fa2f3: Received BeginTabletCopySession request for tablet 866f87ea9f7f48edb667b1b2eb3fa239 from peer 436822a397664c54b8d1b9eb3bc6cff4 ({username='slave'} at 127.25.109.135:37865)
I20240502 01:55:56.911197  6357 tablet_copy_service.cc:161] P 94ffd7aaf85b4e17881241b8396fa2f3: Beginning new tablet copy session on tablet 866f87ea9f7f48edb667b1b2eb3fa239 from peer 436822a397664c54b8d1b9eb3bc6cff4 at {username='slave'} at 127.25.109.135:37865: session id = 436822a397664c54b8d1b9eb3bc6cff4-866f87ea9f7f48edb667b1b2eb3fa239
I20240502 01:55:56.914400  6357 tablet_copy_source_session.cc:205] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:56.916808 10719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 866f87ea9f7f48edb667b1b2eb3fa239. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:56.939241 10719 tablet_copy_client.cc:783] T 866f87ea9f7f48edb667b1b2eb3fa239 P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:56.939828 10719 tablet_copy_client.cc:647] T 866f87ea9f7f48edb667b1b2eb3fa239 P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:56.943131 10719 tablet_copy_client.cc:523] T 866f87ea9f7f48edb667b1b2eb3fa239 P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:56.950428 10719 tablet_bootstrap.cc:492] T 866f87ea9f7f48edb667b1b2eb3fa239 P 436822a397664c54b8d1b9eb3bc6cff4: Bootstrap starting.
I20240502 01:55:56.966773  8406 raft_consensus.cc:1203] T 086509f1ee0a4a81b2f8194b50dbe159 P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 01:55:56.973982  8406 raft_consensus.cc:1203] T 3a7698f6bd4349ae832a22c853d76fea P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 01:55:57.069918 10719 tablet_bootstrap.cc:492] T 866f87ea9f7f48edb667b1b2eb3fa239 P 436822a397664c54b8d1b9eb3bc6cff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:57.070411 10719 tablet_bootstrap.cc:492] T 866f87ea9f7f48edb667b1b2eb3fa239 P 436822a397664c54b8d1b9eb3bc6cff4: Bootstrap complete.
I20240502 01:55:57.070748 10719 ts_tablet_manager.cc:1367] T 866f87ea9f7f48edb667b1b2eb3fa239 P 436822a397664c54b8d1b9eb3bc6cff4: Time spent bootstrapping tablet: real 0.120s	user 0.047s	sys 0.004s
I20240502 01:55:57.071674 10719 raft_consensus.cc:348] T 866f87ea9f7f48edb667b1b2eb3fa239 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:55:57.071970 10719 raft_consensus.cc:729] T 866f87ea9f7f48edb667b1b2eb3fa239 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 436822a397664c54b8d1b9eb3bc6cff4, State: Initialized, Role: FOLLOWER
I20240502 01:55:57.072268 10719 consensus_queue.cc:260] T 866f87ea9f7f48edb667b1b2eb3fa239 P 436822a397664c54b8d1b9eb3bc6cff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:55:57.082090 10719 ts_tablet_manager.cc:1398] T 866f87ea9f7f48edb667b1b2eb3fa239 P 436822a397664c54b8d1b9eb3bc6cff4: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.004s
I20240502 01:55:57.083537  6357 tablet_copy_service.cc:342] P 94ffd7aaf85b4e17881241b8396fa2f3: Request end of tablet copy session 436822a397664c54b8d1b9eb3bc6cff4-866f87ea9f7f48edb667b1b2eb3fa239 received from {username='slave'} at 127.25.109.135:37865
I20240502 01:55:57.083705  6357 tablet_copy_service.cc:433] P 94ffd7aaf85b4e17881241b8396fa2f3: ending tablet copy session 436822a397664c54b8d1b9eb3bc6cff4-866f87ea9f7f48edb667b1b2eb3fa239 on tablet 866f87ea9f7f48edb667b1b2eb3fa239 with peer 436822a397664c54b8d1b9eb3bc6cff4
I20240502 01:55:57.128383  6469 consensus_queue.cc:237] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
W20240502 01:55:57.153918  6401 consensus_peers.cc:485] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 265a9645e1f141cb88c786b9aef8f90e. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:57.154006  6335 raft_consensus.cc:1261] T 265a9645e1f141cb88c786b9aef8f90e P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:57.154650  6068 raft_consensus.cc:1261] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:57.155427 10667 consensus_queue.cc:1035] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:57.156075 10667 consensus_queue.cc:1035] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:57.179252 10480 raft_consensus.cc:2932] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:57.185204  6068 raft_consensus.cc:2932] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:57.189711  6335 raft_consensus.cc:2932] T 265a9645e1f141cb88c786b9aef8f90e P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:57.191660  5917 catalog_manager.cc:5484] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399 reported cstate change: config changed from index -1 to 4, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New cstate: current_term: 1 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:57.384048  8269 raft_consensus.cc:1203] T 866f87ea9f7f48edb667b1b2eb3fa239 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:55:57.395792  6335 consensus_queue.cc:237] T 55de74ae85e541a19008fd2fc2da3ffd P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:55:57.400856  6469 raft_consensus.cc:1261] T 55de74ae85e541a19008fd2fc2da3ffd P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:57.401484 10302 consensus_queue.cc:1035] T 55de74ae85e541a19008fd2fc2da3ffd P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:57.401849  6068 raft_consensus.cc:1261] T 55de74ae85e541a19008fd2fc2da3ffd P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:55:57.402271  6268 consensus_peers.cc:485] T 55de74ae85e541a19008fd2fc2da3ffd P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135:40543): Couldn't send request to peer 436822a397664c54b8d1b9eb3bc6cff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 55de74ae85e541a19008fd2fc2da3ffd. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:57.403239 10302 consensus_queue.cc:1035] T 55de74ae85e541a19008fd2fc2da3ffd P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:57.409000  6068 consensus_queue.cc:237] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:55:57.411499 10302 raft_consensus.cc:2932] T 55de74ae85e541a19008fd2fc2da3ffd P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:55:57.413998  6066 raft_consensus.cc:2932] T 55de74ae85e541a19008fd2fc2da3ffd P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:55:57.419437  6469 raft_consensus.cc:2932] T 55de74ae85e541a19008fd2fc2da3ffd P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
W20240502 01:55:57.420845  5999 consensus_peers.cc:485] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d8a6db6ae11e4cb8881aee95bdb0e59d. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:57.422751  6465 raft_consensus.cc:1261] T d8a6db6ae11e4cb8881aee95bdb0e59d P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:57.424788  6335 raft_consensus.cc:1261] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:57.426456 10689 consensus_queue.cc:1035] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:57.427590 10689 consensus_queue.cc:1035] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:57.437427  5915 catalog_manager.cc:5484] T 55de74ae85e541a19008fd2fc2da3ffd P 07cdb213c77c4ffda1f999b66b725399 reported cstate change: config changed from index -1 to 4, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:55:57.453410  6336 consensus_queue.cc:237] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:55:57.475267  6447 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 01:55:57.489684  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7f2519b0c6404799ad84d120b45a7ea4. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:57.491449  6201 raft_consensus.cc:1261] T 7f2519b0c6404799ad84d120b45a7ea4 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:57.492471 10610 consensus_queue.cc:1035] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:57.499549  6066 raft_consensus.cc:1261] T 7f2519b0c6404799ad84d120b45a7ea4 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:57.502406 10838 consensus_queue.cc:1035] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:57.547327 10689 raft_consensus.cc:2932] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:57.551649  6336 raft_consensus.cc:2932] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:57.555054  6465 raft_consensus.cc:2932] T d8a6db6ae11e4cb8881aee95bdb0e59d P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:57.556221  5915 catalog_manager.cc:5484] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9 reported cstate change: config changed from index -1 to 4, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New cstate: current_term: 1 leader_uuid: "218bc8c1f9ce46059898fc738179adf9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:57.572784  6447 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:55:57.596587  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:57.608824  6046 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:55:57.616828  6046 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:55:57.631691 10858 raft_consensus.cc:2932] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:57.632531  6068 raft_consensus.cc:2932] T 7f2519b0c6404799ad84d120b45a7ea4 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:57.639619  5915 catalog_manager.cc:5484] T 7f2519b0c6404799ad84d120b45a7ea4 P 218bc8c1f9ce46059898fc738179adf9 reported cstate change: config changed from index -1 to 4, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:57.640388  6201 raft_consensus.cc:2932] T 7f2519b0c6404799ad84d120b45a7ea4 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:57.644650  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:57.668542  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:57.668844  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:55:57.691655  8385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:57.706401  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:57.719893 10893 ts_tablet_manager.cc:902] T 265a9645e1f141cb88c786b9aef8f90e P 6d34e50fb5dd4a1db58013d4863b8992: Initiating tablet copy from peer 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867)
I20240502 01:55:57.727380  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:55:57.785313  8385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:57.793908  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:57.833678  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:57.861176  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:57.901691 10920 ts_tablet_manager.cc:902] T 55de74ae85e541a19008fd2fc2da3ffd P 436822a397664c54b8d1b9eb3bc6cff4: Initiating tablet copy from peer 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245)
I20240502 01:55:57.902458  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:57.931740 10929 ts_tablet_manager.cc:902] T d8a6db6ae11e4cb8881aee95bdb0e59d P 6d34e50fb5dd4a1db58013d4863b8992: Initiating tablet copy from peer 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227)
I20240502 01:55:57.935276 10893 tablet_copy_client.cc:308] T 265a9645e1f141cb88c786b9aef8f90e P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.132:36867
I20240502 01:55:57.936964  6489 tablet_copy_service.cc:140] P 07cdb213c77c4ffda1f999b66b725399: Received BeginTabletCopySession request for tablet 265a9645e1f141cb88c786b9aef8f90e from peer 6d34e50fb5dd4a1db58013d4863b8992 ({username='slave'} at 127.25.109.136:34855)
I20240502 01:55:57.937186  6489 tablet_copy_service.cc:161] P 07cdb213c77c4ffda1f999b66b725399: Beginning new tablet copy session on tablet 265a9645e1f141cb88c786b9aef8f90e from peer 6d34e50fb5dd4a1db58013d4863b8992 at {username='slave'} at 127.25.109.136:34855: session id = 6d34e50fb5dd4a1db58013d4863b8992-265a9645e1f141cb88c786b9aef8f90e
I20240502 01:55:57.940567  6489 tablet_copy_source_session.cc:205] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:57.942065 10893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 265a9645e1f141cb88c786b9aef8f90e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:57.947906  6066 consensus_queue.cc:237] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
W20240502 01:55:57.953954  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4f4088f3523f4cd0afdf0de0f447c7af. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:57.954285  6336 raft_consensus.cc:1261] T 4f4088f3523f4cd0afdf0de0f447c7af P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:57.955224  6201 raft_consensus.cc:1261] T 4f4088f3523f4cd0afdf0de0f447c7af P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:57.955327 10691 consensus_queue.cc:1035] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:57.957399 10691 consensus_queue.cc:1035] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:57.978583 10691 raft_consensus.cc:2932] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:57.985050  6336 raft_consensus.cc:2932] T 4f4088f3523f4cd0afdf0de0f447c7af P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:57.986471  6201 raft_consensus.cc:2932] T 4f4088f3523f4cd0afdf0de0f447c7af P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:57.989022  5915 catalog_manager.cc:5484] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 reported cstate change: config changed from index -1 to 4, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New cstate: current_term: 1 leader_uuid: "218bc8c1f9ce46059898fc738179adf9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:55:58.036242 10893 tablet_copy_client.cc:783] T 265a9645e1f141cb88c786b9aef8f90e P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:58.037565 10893 tablet_copy_client.cc:647] T 265a9645e1f141cb88c786b9aef8f90e P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:58.044157 10920 tablet_copy_client.cc:308] T 55de74ae85e541a19008fd2fc2da3ffd P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.131:44245
I20240502 01:55:58.044479 10893 tablet_copy_client.cc:523] T 265a9645e1f141cb88c786b9aef8f90e P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:58.045217  6357 tablet_copy_service.cc:140] P 94ffd7aaf85b4e17881241b8396fa2f3: Received BeginTabletCopySession request for tablet 55de74ae85e541a19008fd2fc2da3ffd from peer 436822a397664c54b8d1b9eb3bc6cff4 ({username='slave'} at 127.25.109.135:37865)
I20240502 01:55:58.045411  6357 tablet_copy_service.cc:161] P 94ffd7aaf85b4e17881241b8396fa2f3: Beginning new tablet copy session on tablet 55de74ae85e541a19008fd2fc2da3ffd from peer 436822a397664c54b8d1b9eb3bc6cff4 at {username='slave'} at 127.25.109.135:37865: session id = 436822a397664c54b8d1b9eb3bc6cff4-55de74ae85e541a19008fd2fc2da3ffd
I20240502 01:55:58.047837 10950 ts_tablet_manager.cc:902] T 7f2519b0c6404799ad84d120b45a7ea4 P 9db17f84d37944098cbeb9e0d8456efb: Initiating tablet copy from peer 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245)
I20240502 01:55:58.048460  6357 tablet_copy_source_session.cc:205] T 55de74ae85e541a19008fd2fc2da3ffd P 94ffd7aaf85b4e17881241b8396fa2f3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:58.050205 10920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55de74ae85e541a19008fd2fc2da3ffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:58.050594 10929 tablet_copy_client.cc:308] T d8a6db6ae11e4cb8881aee95bdb0e59d P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.129:36227
I20240502 01:55:58.051352  6089 tablet_copy_service.cc:140] P 218bc8c1f9ce46059898fc738179adf9: Received BeginTabletCopySession request for tablet d8a6db6ae11e4cb8881aee95bdb0e59d from peer 6d34e50fb5dd4a1db58013d4863b8992 ({username='slave'} at 127.25.109.136:45167)
I20240502 01:55:58.051556  6089 tablet_copy_service.cc:161] P 218bc8c1f9ce46059898fc738179adf9: Beginning new tablet copy session on tablet d8a6db6ae11e4cb8881aee95bdb0e59d from peer 6d34e50fb5dd4a1db58013d4863b8992 at {username='slave'} at 127.25.109.136:45167: session id = 6d34e50fb5dd4a1db58013d4863b8992-d8a6db6ae11e4cb8881aee95bdb0e59d
I20240502 01:55:58.054677  6089 tablet_copy_source_session.cc:205] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:58.057780 10929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8a6db6ae11e4cb8881aee95bdb0e59d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:58.058285 10920 tablet_copy_client.cc:783] T 55de74ae85e541a19008fd2fc2da3ffd P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:58.058595 10920 tablet_copy_client.cc:647] T 55de74ae85e541a19008fd2fc2da3ffd P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:58.060518 10920 tablet_copy_client.cc:523] T 55de74ae85e541a19008fd2fc2da3ffd P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:58.064182 10920 tablet_bootstrap.cc:492] T 55de74ae85e541a19008fd2fc2da3ffd P 436822a397664c54b8d1b9eb3bc6cff4: Bootstrap starting.
I20240502 01:55:58.070586 10893 tablet_bootstrap.cc:492] T 265a9645e1f141cb88c786b9aef8f90e P 6d34e50fb5dd4a1db58013d4863b8992: Bootstrap starting.
I20240502 01:55:58.103581 10950 tablet_copy_client.cc:308] T 7f2519b0c6404799ad84d120b45a7ea4 P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.131:44245
I20240502 01:55:58.104449  6357 tablet_copy_service.cc:140] P 94ffd7aaf85b4e17881241b8396fa2f3: Received BeginTabletCopySession request for tablet 7f2519b0c6404799ad84d120b45a7ea4 from peer 9db17f84d37944098cbeb9e0d8456efb ({username='slave'} at 127.25.109.134:41767)
I20240502 01:55:58.104629  6357 tablet_copy_service.cc:161] P 94ffd7aaf85b4e17881241b8396fa2f3: Beginning new tablet copy session on tablet 7f2519b0c6404799ad84d120b45a7ea4 from peer 9db17f84d37944098cbeb9e0d8456efb at {username='slave'} at 127.25.109.134:41767: session id = 9db17f84d37944098cbeb9e0d8456efb-7f2519b0c6404799ad84d120b45a7ea4
I20240502 01:55:58.107631  6357 tablet_copy_source_session.cc:205] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:58.114202 10950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f2519b0c6404799ad84d120b45a7ea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:58.135970 10929 tablet_copy_client.cc:783] T d8a6db6ae11e4cb8881aee95bdb0e59d P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:58.139642 10929 tablet_copy_client.cc:647] T d8a6db6ae11e4cb8881aee95bdb0e59d P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:58.146849 10929 tablet_copy_client.cc:523] T d8a6db6ae11e4cb8881aee95bdb0e59d P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:58.154687  6465 consensus_queue.cc:237] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:55:58.158298 10950 tablet_copy_client.cc:783] T 7f2519b0c6404799ad84d120b45a7ea4 P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Starting download of 0 data blocks...
W20240502 01:55:58.158409  6401 consensus_peers.cc:485] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:58.158609 10950 tablet_copy_client.cc:647] T 7f2519b0c6404799ad84d120b45a7ea4 P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:58.159641  6068 raft_consensus.cc:1261] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 01:55:58.160451  6402 consensus_peers.cc:485] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 265a9645e1f141cb88c786b9aef8f90e. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:58.161125 10667 consensus_queue.cc:1035] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:58.172855  6336 raft_consensus.cc:1261] T 265a9645e1f141cb88c786b9aef8f90e P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:58.173605 10667 consensus_queue.cc:1035] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:58.176743 10480 raft_consensus.cc:2932] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:58.182571  6336 raft_consensus.cc:2932] T 265a9645e1f141cb88c786b9aef8f90e P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:58.191819  6068 raft_consensus.cc:2932] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:58.199672  5921 catalog_manager.cc:5484] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 reported cstate change: config changed from index 4 to 5, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New cstate: current_term: 1 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:58.210500 10920 tablet_bootstrap.cc:492] T 55de74ae85e541a19008fd2fc2da3ffd P 436822a397664c54b8d1b9eb3bc6cff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:58.211607 10920 tablet_bootstrap.cc:492] T 55de74ae85e541a19008fd2fc2da3ffd P 436822a397664c54b8d1b9eb3bc6cff4: Bootstrap complete.
I20240502 01:55:58.153034 10929 tablet_bootstrap.cc:492] T d8a6db6ae11e4cb8881aee95bdb0e59d P 6d34e50fb5dd4a1db58013d4863b8992: Bootstrap starting.
I20240502 01:55:58.212088 10920 ts_tablet_manager.cc:1367] T 55de74ae85e541a19008fd2fc2da3ffd P 436822a397664c54b8d1b9eb3bc6cff4: Time spent bootstrapping tablet: real 0.148s	user 0.070s	sys 0.029s
I20240502 01:55:58.212975 10920 raft_consensus.cc:348] T 55de74ae85e541a19008fd2fc2da3ffd P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:55:58.213380 10920 raft_consensus.cc:729] T 55de74ae85e541a19008fd2fc2da3ffd P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 436822a397664c54b8d1b9eb3bc6cff4, State: Initialized, Role: FOLLOWER
I20240502 01:55:58.213660 10920 consensus_queue.cc:260] T 55de74ae85e541a19008fd2fc2da3ffd P 436822a397664c54b8d1b9eb3bc6cff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:55:58.215703 10950 tablet_copy_client.cc:523] T 7f2519b0c6404799ad84d120b45a7ea4 P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:58.252640 10950 tablet_bootstrap.cc:492] T 7f2519b0c6404799ad84d120b45a7ea4 P 9db17f84d37944098cbeb9e0d8456efb: Bootstrap starting.
I20240502 01:55:58.288100 10920 ts_tablet_manager.cc:1398] T 55de74ae85e541a19008fd2fc2da3ffd P 436822a397664c54b8d1b9eb3bc6cff4: Time spent starting tablet: real 0.076s	user 0.023s	sys 0.029s
I20240502 01:55:58.289606  6357 tablet_copy_service.cc:342] P 94ffd7aaf85b4e17881241b8396fa2f3: Request end of tablet copy session 436822a397664c54b8d1b9eb3bc6cff4-55de74ae85e541a19008fd2fc2da3ffd received from {username='slave'} at 127.25.109.135:37865
I20240502 01:55:58.289790  6357 tablet_copy_service.cc:433] P 94ffd7aaf85b4e17881241b8396fa2f3: ending tablet copy session 436822a397664c54b8d1b9eb3bc6cff4-55de74ae85e541a19008fd2fc2da3ffd on tablet 55de74ae85e541a19008fd2fc2da3ffd with peer 436822a397664c54b8d1b9eb3bc6cff4
I20240502 01:55:58.338668  8269 raft_consensus.cc:1203] T 55de74ae85e541a19008fd2fc2da3ffd P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:55:58.391333 11025 ts_tablet_manager.cc:902] T 4f4088f3523f4cd0afdf0de0f447c7af P 6d34e50fb5dd4a1db58013d4863b8992: Initiating tablet copy from peer 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227)
I20240502 01:55:58.398960 10893 tablet_bootstrap.cc:492] T 265a9645e1f141cb88c786b9aef8f90e P 6d34e50fb5dd4a1db58013d4863b8992: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:58.399453 10893 tablet_bootstrap.cc:492] T 265a9645e1f141cb88c786b9aef8f90e P 6d34e50fb5dd4a1db58013d4863b8992: Bootstrap complete.
I20240502 01:55:58.399780 10893 ts_tablet_manager.cc:1367] T 265a9645e1f141cb88c786b9aef8f90e P 6d34e50fb5dd4a1db58013d4863b8992: Time spent bootstrapping tablet: real 0.329s	user 0.045s	sys 0.004s
I20240502 01:55:58.402451 10893 raft_consensus.cc:348] T 265a9645e1f141cb88c786b9aef8f90e P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:55:58.403967 10893 raft_consensus.cc:729] T 265a9645e1f141cb88c786b9aef8f90e P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d34e50fb5dd4a1db58013d4863b8992, State: Initialized, Role: FOLLOWER
I20240502 01:55:58.404693 10893 consensus_queue.cc:260] T 265a9645e1f141cb88c786b9aef8f90e P 6d34e50fb5dd4a1db58013d4863b8992 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:55:58.413887 11025 tablet_copy_client.cc:308] T 4f4088f3523f4cd0afdf0de0f447c7af P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.129:36227
I20240502 01:55:58.414963  6089 tablet_copy_service.cc:140] P 218bc8c1f9ce46059898fc738179adf9: Received BeginTabletCopySession request for tablet 4f4088f3523f4cd0afdf0de0f447c7af from peer 6d34e50fb5dd4a1db58013d4863b8992 ({username='slave'} at 127.25.109.136:45167)
I20240502 01:55:58.415136  6089 tablet_copy_service.cc:161] P 218bc8c1f9ce46059898fc738179adf9: Beginning new tablet copy session on tablet 4f4088f3523f4cd0afdf0de0f447c7af from peer 6d34e50fb5dd4a1db58013d4863b8992 at {username='slave'} at 127.25.109.136:45167: session id = 6d34e50fb5dd4a1db58013d4863b8992-4f4088f3523f4cd0afdf0de0f447c7af
I20240502 01:55:58.418323  6089 tablet_copy_source_session.cc:205] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:58.420573 11025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f4088f3523f4cd0afdf0de0f447c7af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:58.427228 11025 tablet_copy_client.cc:783] T 4f4088f3523f4cd0afdf0de0f447c7af P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:58.427520 11025 tablet_copy_client.cc:647] T 4f4088f3523f4cd0afdf0de0f447c7af P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:58.429494 11025 tablet_copy_client.cc:523] T 4f4088f3523f4cd0afdf0de0f447c7af P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:58.432714 10929 tablet_bootstrap.cc:492] T d8a6db6ae11e4cb8881aee95bdb0e59d P 6d34e50fb5dd4a1db58013d4863b8992: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:58.433147 10929 tablet_bootstrap.cc:492] T d8a6db6ae11e4cb8881aee95bdb0e59d P 6d34e50fb5dd4a1db58013d4863b8992: Bootstrap complete.
I20240502 01:55:58.433195 11025 tablet_bootstrap.cc:492] T 4f4088f3523f4cd0afdf0de0f447c7af P 6d34e50fb5dd4a1db58013d4863b8992: Bootstrap starting.
I20240502 01:55:58.433465 10929 ts_tablet_manager.cc:1367] T d8a6db6ae11e4cb8881aee95bdb0e59d P 6d34e50fb5dd4a1db58013d4863b8992: Time spent bootstrapping tablet: real 0.281s	user 0.047s	sys 0.005s
I20240502 01:55:58.412437 10893 ts_tablet_manager.cc:1398] T 265a9645e1f141cb88c786b9aef8f90e P 6d34e50fb5dd4a1db58013d4863b8992: Time spent starting tablet: real 0.011s	user 0.005s	sys 0.000s
I20240502 01:55:58.434445  6489 tablet_copy_service.cc:342] P 07cdb213c77c4ffda1f999b66b725399: Request end of tablet copy session 6d34e50fb5dd4a1db58013d4863b8992-265a9645e1f141cb88c786b9aef8f90e received from {username='slave'} at 127.25.109.136:34855
I20240502 01:55:58.434577  6489 tablet_copy_service.cc:433] P 07cdb213c77c4ffda1f999b66b725399: ending tablet copy session 6d34e50fb5dd4a1db58013d4863b8992-265a9645e1f141cb88c786b9aef8f90e on tablet 265a9645e1f141cb88c786b9aef8f90e with peer 6d34e50fb5dd4a1db58013d4863b8992
I20240502 01:55:58.434468 10950 tablet_bootstrap.cc:492] T 7f2519b0c6404799ad84d120b45a7ea4 P 9db17f84d37944098cbeb9e0d8456efb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:58.435029 10950 tablet_bootstrap.cc:492] T 7f2519b0c6404799ad84d120b45a7ea4 P 9db17f84d37944098cbeb9e0d8456efb: Bootstrap complete.
I20240502 01:55:58.435302 10950 ts_tablet_manager.cc:1367] T 7f2519b0c6404799ad84d120b45a7ea4 P 9db17f84d37944098cbeb9e0d8456efb: Time spent bootstrapping tablet: real 0.216s	user 0.050s	sys 0.004s
I20240502 01:55:58.437315 10950 raft_consensus.cc:348] T 7f2519b0c6404799ad84d120b45a7ea4 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:55:58.437575 10950 raft_consensus.cc:729] T 7f2519b0c6404799ad84d120b45a7ea4 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9db17f84d37944098cbeb9e0d8456efb, State: Initialized, Role: FOLLOWER
I20240502 01:55:58.437992 10950 consensus_queue.cc:260] T 7f2519b0c6404799ad84d120b45a7ea4 P 9db17f84d37944098cbeb9e0d8456efb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:55:58.439967 10929 raft_consensus.cc:348] T d8a6db6ae11e4cb8881aee95bdb0e59d P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:55:58.440256 10929 raft_consensus.cc:729] T d8a6db6ae11e4cb8881aee95bdb0e59d P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d34e50fb5dd4a1db58013d4863b8992, State: Initialized, Role: FOLLOWER
I20240502 01:55:58.440559 10929 consensus_queue.cc:260] T d8a6db6ae11e4cb8881aee95bdb0e59d P 6d34e50fb5dd4a1db58013d4863b8992 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:55:58.443691 10929 ts_tablet_manager.cc:1398] T d8a6db6ae11e4cb8881aee95bdb0e59d P 6d34e50fb5dd4a1db58013d4863b8992: Time spent starting tablet: real 0.010s	user 0.003s	sys 0.000s
I20240502 01:55:58.445833 10950 ts_tablet_manager.cc:1398] T 7f2519b0c6404799ad84d120b45a7ea4 P 9db17f84d37944098cbeb9e0d8456efb: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.003s
I20240502 01:55:58.447782  6357 tablet_copy_service.cc:342] P 94ffd7aaf85b4e17881241b8396fa2f3: Request end of tablet copy session 9db17f84d37944098cbeb9e0d8456efb-7f2519b0c6404799ad84d120b45a7ea4 received from {username='slave'} at 127.25.109.134:41767
I20240502 01:55:58.448036  6357 tablet_copy_service.cc:433] P 94ffd7aaf85b4e17881241b8396fa2f3: ending tablet copy session 9db17f84d37944098cbeb9e0d8456efb-7f2519b0c6404799ad84d120b45a7ea4 on tablet 7f2519b0c6404799ad84d120b45a7ea4 with peer 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:55:58.452659  6089 tablet_copy_service.cc:342] P 218bc8c1f9ce46059898fc738179adf9: Request end of tablet copy session 6d34e50fb5dd4a1db58013d4863b8992-d8a6db6ae11e4cb8881aee95bdb0e59d received from {username='slave'} at 127.25.109.136:45167
I20240502 01:55:58.452857  6089 tablet_copy_service.cc:433] P 218bc8c1f9ce46059898fc738179adf9: ending tablet copy session 6d34e50fb5dd4a1db58013d4863b8992-d8a6db6ae11e4cb8881aee95bdb0e59d on tablet d8a6db6ae11e4cb8881aee95bdb0e59d with peer 6d34e50fb5dd4a1db58013d4863b8992
I20240502 01:55:58.476042  8135 raft_consensus.cc:1203] T 7f2519b0c6404799ad84d120b45a7ea4 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 01:55:58.488179  8406 raft_consensus.cc:1203] T d8a6db6ae11e4cb8881aee95bdb0e59d P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
W20240502 01:55:58.494783  6134 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.135:40543, user_credentials={real_user=slave}} blocked reactor thread for 46075.1us
I20240502 01:55:58.558372 11025 tablet_bootstrap.cc:492] T 4f4088f3523f4cd0afdf0de0f447c7af P 6d34e50fb5dd4a1db58013d4863b8992: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:58.564466 11025 tablet_bootstrap.cc:492] T 4f4088f3523f4cd0afdf0de0f447c7af P 6d34e50fb5dd4a1db58013d4863b8992: Bootstrap complete.
I20240502 01:55:58.564976 11025 ts_tablet_manager.cc:1367] T 4f4088f3523f4cd0afdf0de0f447c7af P 6d34e50fb5dd4a1db58013d4863b8992: Time spent bootstrapping tablet: real 0.132s	user 0.041s	sys 0.016s
I20240502 01:55:58.566198 11025 raft_consensus.cc:348] T 4f4088f3523f4cd0afdf0de0f447c7af P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:55:58.566933 11025 raft_consensus.cc:729] T 4f4088f3523f4cd0afdf0de0f447c7af P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d34e50fb5dd4a1db58013d4863b8992, State: Initialized, Role: FOLLOWER
I20240502 01:55:58.567932 11025 consensus_queue.cc:260] T 4f4088f3523f4cd0afdf0de0f447c7af P 6d34e50fb5dd4a1db58013d4863b8992 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:55:58.568594 11025 ts_tablet_manager.cc:1398] T 4f4088f3523f4cd0afdf0de0f447c7af P 6d34e50fb5dd4a1db58013d4863b8992: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 01:55:58.570096  6089 tablet_copy_service.cc:342] P 218bc8c1f9ce46059898fc738179adf9: Request end of tablet copy session 6d34e50fb5dd4a1db58013d4863b8992-4f4088f3523f4cd0afdf0de0f447c7af received from {username='slave'} at 127.25.109.136:45167
I20240502 01:55:58.570268  6089 tablet_copy_service.cc:433] P 218bc8c1f9ce46059898fc738179adf9: ending tablet copy session 6d34e50fb5dd4a1db58013d4863b8992-4f4088f3523f4cd0afdf0de0f447c7af on tablet 4f4088f3523f4cd0afdf0de0f447c7af with peer 6d34e50fb5dd4a1db58013d4863b8992
I20240502 01:55:58.640908 10950 ts_tablet_manager.cc:902] T 265a9645e1f141cb88c786b9aef8f90e P 9db17f84d37944098cbeb9e0d8456efb: Initiating tablet copy from peer 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867)
I20240502 01:55:58.691740 10950 tablet_copy_client.cc:308] T 265a9645e1f141cb88c786b9aef8f90e P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.132:36867
I20240502 01:55:58.692726  6489 tablet_copy_service.cc:140] P 07cdb213c77c4ffda1f999b66b725399: Received BeginTabletCopySession request for tablet 265a9645e1f141cb88c786b9aef8f90e from peer 9db17f84d37944098cbeb9e0d8456efb ({username='slave'} at 127.25.109.134:56199)
I20240502 01:55:58.692938  6489 tablet_copy_service.cc:161] P 07cdb213c77c4ffda1f999b66b725399: Beginning new tablet copy session on tablet 265a9645e1f141cb88c786b9aef8f90e from peer 9db17f84d37944098cbeb9e0d8456efb at {username='slave'} at 127.25.109.134:56199: session id = 9db17f84d37944098cbeb9e0d8456efb-265a9645e1f141cb88c786b9aef8f90e
I20240502 01:55:58.696727  6489 tablet_copy_source_session.cc:205] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:58.699925 10950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 265a9645e1f141cb88c786b9aef8f90e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:58.731022  8406 raft_consensus.cc:2932] T 265a9645e1f141cb88c786b9aef8f90e P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:58.748234 10950 tablet_copy_client.cc:783] T 265a9645e1f141cb88c786b9aef8f90e P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:58.748880 10950 tablet_copy_client.cc:647] T 265a9645e1f141cb88c786b9aef8f90e P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:58.751937 10950 tablet_copy_client.cc:523] T 265a9645e1f141cb88c786b9aef8f90e P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:58.759557 10950 tablet_bootstrap.cc:492] T 265a9645e1f141cb88c786b9aef8f90e P 9db17f84d37944098cbeb9e0d8456efb: Bootstrap starting.
I20240502 01:55:58.766448  6068 consensus_queue.cc:237] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:55:58.772621  8406 raft_consensus.cc:1261] T 4f4088f3523f4cd0afdf0de0f447c7af P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:58.773449  6201 raft_consensus.cc:1261] T 4f4088f3523f4cd0afdf0de0f447c7af P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:58.774194 10691 consensus_queue.cc:1035] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:58.775375 10691 consensus_queue.cc:1035] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 01:55:58.777630  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4f4088f3523f4cd0afdf0de0f447c7af. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:58.819314  6335 raft_consensus.cc:1261] T 4f4088f3523f4cd0afdf0de0f447c7af P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:58.824951 10691 consensus_queue.cc:1035] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:58.825492 10691 raft_consensus.cc:2932] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:58.829432  8406 raft_consensus.cc:2932] T 4f4088f3523f4cd0afdf0de0f447c7af P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:58.834574  6335 raft_consensus.cc:2932] T 4f4088f3523f4cd0afdf0de0f447c7af P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:58.856966  5912 catalog_manager.cc:5484] T 4f4088f3523f4cd0afdf0de0f447c7af P 6d34e50fb5dd4a1db58013d4863b8992 reported cstate change: config changed from index 4 to 5, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New cstate: current_term: 1 leader_uuid: "218bc8c1f9ce46059898fc738179adf9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:58.861390 10950 tablet_bootstrap.cc:492] T 265a9645e1f141cb88c786b9aef8f90e P 9db17f84d37944098cbeb9e0d8456efb: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:58.861958 10950 tablet_bootstrap.cc:492] T 265a9645e1f141cb88c786b9aef8f90e P 9db17f84d37944098cbeb9e0d8456efb: Bootstrap complete.
I20240502 01:55:58.863046 10950 ts_tablet_manager.cc:1367] T 265a9645e1f141cb88c786b9aef8f90e P 9db17f84d37944098cbeb9e0d8456efb: Time spent bootstrapping tablet: real 0.104s	user 0.072s	sys 0.020s
I20240502 01:55:58.864034 10950 raft_consensus.cc:348] T 265a9645e1f141cb88c786b9aef8f90e P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:55:58.864399 10950 raft_consensus.cc:729] T 265a9645e1f141cb88c786b9aef8f90e P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9db17f84d37944098cbeb9e0d8456efb, State: Initialized, Role: FOLLOWER
I20240502 01:55:58.864722 10950 consensus_queue.cc:260] T 265a9645e1f141cb88c786b9aef8f90e P 9db17f84d37944098cbeb9e0d8456efb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:55:58.865352 10950 ts_tablet_manager.cc:1398] T 265a9645e1f141cb88c786b9aef8f90e P 9db17f84d37944098cbeb9e0d8456efb: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 01:55:58.866112  6489 tablet_copy_service.cc:342] P 07cdb213c77c4ffda1f999b66b725399: Request end of tablet copy session 9db17f84d37944098cbeb9e0d8456efb-265a9645e1f141cb88c786b9aef8f90e received from {username='slave'} at 127.25.109.134:56199
I20240502 01:55:58.866292  6489 tablet_copy_service.cc:433] P 07cdb213c77c4ffda1f999b66b725399: ending tablet copy session 9db17f84d37944098cbeb9e0d8456efb-265a9645e1f141cb88c786b9aef8f90e on tablet 265a9645e1f141cb88c786b9aef8f90e with peer 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:55:58.876397  6204 raft_consensus.cc:2932] T 4f4088f3523f4cd0afdf0de0f447c7af P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:55:58.887068  6046 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:55:58.906739  6447 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:55:59.001647  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:59.056411  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:59.077790  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:55:59.115276  8385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:59.124500  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:59.136557  8135 raft_consensus.cc:1203] T 265a9645e1f141cb88c786b9aef8f90e P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 01:55:59.172582  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:59.310580 10950 ts_tablet_manager.cc:902] T 4f4088f3523f4cd0afdf0de0f447c7af P 9db17f84d37944098cbeb9e0d8456efb: Initiating tablet copy from peer 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227)
I20240502 01:55:59.334065  6046 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:55:59.378487  6447 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:55:59.392498  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:59.397426  8385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:59.407469  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:55:59.423131  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:59.441100  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:59.446122 10950 tablet_copy_client.cc:308] T 4f4088f3523f4cd0afdf0de0f447c7af P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.129:36227
I20240502 01:55:59.446867  6089 tablet_copy_service.cc:140] P 218bc8c1f9ce46059898fc738179adf9: Received BeginTabletCopySession request for tablet 4f4088f3523f4cd0afdf0de0f447c7af from peer 9db17f84d37944098cbeb9e0d8456efb ({username='slave'} at 127.25.109.134:39709)
I20240502 01:55:59.447090  6089 tablet_copy_service.cc:161] P 218bc8c1f9ce46059898fc738179adf9: Beginning new tablet copy session on tablet 4f4088f3523f4cd0afdf0de0f447c7af from peer 9db17f84d37944098cbeb9e0d8456efb at {username='slave'} at 127.25.109.134:39709: session id = 9db17f84d37944098cbeb9e0d8456efb-4f4088f3523f4cd0afdf0de0f447c7af
I20240502 01:55:59.450145  6089 tablet_copy_source_session.cc:205] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:59.454419 10950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f4088f3523f4cd0afdf0de0f447c7af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:59.459997  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:59.476995  8385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:59.483188  6046 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:55:59.496824 10950 tablet_copy_client.cc:783] T 4f4088f3523f4cd0afdf0de0f447c7af P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:59.497280 10950 tablet_copy_client.cc:647] T 4f4088f3523f4cd0afdf0de0f447c7af P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:59.502121  6447 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:55:59.505985 10950 tablet_copy_client.cc:523] T 4f4088f3523f4cd0afdf0de0f447c7af P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:59.521497  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:59.526690 10950 tablet_bootstrap.cc:492] T 4f4088f3523f4cd0afdf0de0f447c7af P 9db17f84d37944098cbeb9e0d8456efb: Bootstrap starting.
I20240502 01:55:59.535740  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:55:59.596946  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:59.633596  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:59.638654  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:59.830287 10950 tablet_bootstrap.cc:492] T 4f4088f3523f4cd0afdf0de0f447c7af P 9db17f84d37944098cbeb9e0d8456efb: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:59.830785 10950 tablet_bootstrap.cc:492] T 4f4088f3523f4cd0afdf0de0f447c7af P 9db17f84d37944098cbeb9e0d8456efb: Bootstrap complete.
I20240502 01:55:59.831178 10950 ts_tablet_manager.cc:1367] T 4f4088f3523f4cd0afdf0de0f447c7af P 9db17f84d37944098cbeb9e0d8456efb: Time spent bootstrapping tablet: real 0.305s	user 0.051s	sys 0.005s
I20240502 01:55:59.832617 10950 raft_consensus.cc:348] T 4f4088f3523f4cd0afdf0de0f447c7af P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:55:59.832998 10950 raft_consensus.cc:729] T 4f4088f3523f4cd0afdf0de0f447c7af P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9db17f84d37944098cbeb9e0d8456efb, State: Initialized, Role: FOLLOWER
I20240502 01:55:59.833319 10950 consensus_queue.cc:260] T 4f4088f3523f4cd0afdf0de0f447c7af P 9db17f84d37944098cbeb9e0d8456efb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:55:59.860342 10950 ts_tablet_manager.cc:1398] T 4f4088f3523f4cd0afdf0de0f447c7af P 9db17f84d37944098cbeb9e0d8456efb: Time spent starting tablet: real 0.029s	user 0.005s	sys 0.000s
I20240502 01:55:59.861572  6089 tablet_copy_service.cc:342] P 218bc8c1f9ce46059898fc738179adf9: Request end of tablet copy session 9db17f84d37944098cbeb9e0d8456efb-4f4088f3523f4cd0afdf0de0f447c7af received from {username='slave'} at 127.25.109.134:39709
I20240502 01:55:59.861740  6089 tablet_copy_service.cc:433] P 218bc8c1f9ce46059898fc738179adf9: ending tablet copy session 9db17f84d37944098cbeb9e0d8456efb-4f4088f3523f4cd0afdf0de0f447c7af on tablet 4f4088f3523f4cd0afdf0de0f447c7af with peer 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:55:59.977046  6447 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:55:59.988726  6046 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:00.034665  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:00.073442  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:00.077956  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:00.083850  8385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:00.121996  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:56:00.147831  8135 raft_consensus.cc:1203] T 4f4088f3523f4cd0afdf0de0f447c7af P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 01:56:00.158625  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:00.571771  6447 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:56:00.623164  6046 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:00.676909  8385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:00.698675  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:56:00.725759  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:00.765576  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:00.785384  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:00.792356  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:00.839242  6046 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:00.893333  6447 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:56:00.910679  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:00.950594  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:01.001237  8385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:01.039611  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:56:01.044034  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:01.074311  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:01.419029  8385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:01.429605  6447 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:56:01.432763  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:01.498317  6046 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:01.533828  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:01.542862  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:56:01.622397  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:01.623879  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:01.703406  6447 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:56:01.761703  8385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:01.782815  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:01.792822  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:01.836454  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:01.837638  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:01.863970  6204 consensus_queue.cc:237] T e4d4a1bd80f44b668a2d504e47fb744a P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:56:01.875270  6046 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:01.875416  6048 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:01.880601  8270 raft_consensus.cc:1261] T e4d4a1bd80f44b668a2d504e47fb744a P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Refusing update from remote peer a2c90ee440b44b708bccc2e2d5385cc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:56:01.881441 11397 consensus_queue.cc:1035] T e4d4a1bd80f44b668a2d504e47fb744a P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:56:01.883173  6469 raft_consensus.cc:1261] T e4d4a1bd80f44b668a2d504e47fb744a P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Refusing update from remote peer a2c90ee440b44b708bccc2e2d5385cc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:56:01.884248 11397 consensus_queue.cc:1035] T e4d4a1bd80f44b668a2d504e47fb744a P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:56:01.924374  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:01.943643 11578 raft_consensus.cc:2932] T e4d4a1bd80f44b668a2d504e47fb744a P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:01.947662  8269 raft_consensus.cc:2932] T e4d4a1bd80f44b668a2d504e47fb744a P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:01.953938  5914 catalog_manager.cc:5484] T e4d4a1bd80f44b668a2d504e47fb744a P a2c90ee440b44b708bccc2e2d5385cc0 reported cstate change: config changed from index 4 to 5, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New cstate: current_term: 1 leader_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:01.973107  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:01.976992  8385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:01.980592  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:56:02.004268  6068 consensus_queue.cc:237] T a398da35f18c4df394db82c5bf89aeb1 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:56:02.005084  6465 raft_consensus.cc:2932] T e4d4a1bd80f44b668a2d504e47fb744a P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:02.006934  6447 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:56:02.031167  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:56:02.031895  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:02.052640  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:02.061851  8269 raft_consensus.cc:1261] T a398da35f18c4df394db82c5bf89aeb1 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:56:02.062706 11443 consensus_queue.cc:1035] T a398da35f18c4df394db82c5bf89aeb1 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:56:02.073447  6335 raft_consensus.cc:1261] T a398da35f18c4df394db82c5bf89aeb1 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:56:02.074247 11443 consensus_queue.cc:1035] T a398da35f18c4df394db82c5bf89aeb1 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:56:02.083881 11443 raft_consensus.cc:2932] T a398da35f18c4df394db82c5bf89aeb1 P 218bc8c1f9ce46059898fc738179adf9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:02.092378  8269 raft_consensus.cc:2932] T a398da35f18c4df394db82c5bf89aeb1 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:02.107795  5916 catalog_manager.cc:5484] T a398da35f18c4df394db82c5bf89aeb1 P 218bc8c1f9ce46059898fc738179adf9 reported cstate change: config changed from index 4 to 5, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New cstate: current_term: 1 leader_uuid: "218bc8c1f9ce46059898fc738179adf9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:02.148548  6335 raft_consensus.cc:2932] T a398da35f18c4df394db82c5bf89aeb1 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:02.176146  6180 tablet_service.cc:1514] Processing DeleteTablet for tablet a398da35f18c4df394db82c5bf89aeb1 with delete_type TABLET_DATA_TOMBSTONED (TS a2c90ee440b44b708bccc2e2d5385cc0 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41258
I20240502 01:56:02.177486  6316 tablet_service.cc:1514] Processing DeleteTablet for tablet e4d4a1bd80f44b668a2d504e47fb744a with delete_type TABLET_DATA_TOMBSTONED (TS 94ffd7aaf85b4e17881241b8396fa2f3 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41134
I20240502 01:56:02.193404 11602 tablet_replica.cc:330] T e4d4a1bd80f44b668a2d504e47fb744a P 94ffd7aaf85b4e17881241b8396fa2f3: stopping tablet replica
I20240502 01:56:02.194104 11602 raft_consensus.cc:2229] T e4d4a1bd80f44b668a2d504e47fb744a P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:02.194636 11602 raft_consensus.cc:2258] T e4d4a1bd80f44b668a2d504e47fb744a P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:02.206248 11602 ts_tablet_manager.cc:1875] T e4d4a1bd80f44b668a2d504e47fb744a P 94ffd7aaf85b4e17881241b8396fa2f3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:56:02.216984 11601 tablet_replica.cc:330] T a398da35f18c4df394db82c5bf89aeb1 P a2c90ee440b44b708bccc2e2d5385cc0: stopping tablet replica
I20240502 01:56:02.217730 11601 raft_consensus.cc:2229] T a398da35f18c4df394db82c5bf89aeb1 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:02.218231 11601 raft_consensus.cc:2258] T a398da35f18c4df394db82c5bf89aeb1 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:02.230417 11602 ts_tablet_manager.cc:1888] T e4d4a1bd80f44b668a2d504e47fb744a P 94ffd7aaf85b4e17881241b8396fa2f3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:56:02.230712 11602 log.cc:1196] T e4d4a1bd80f44b668a2d504e47fb744a P 94ffd7aaf85b4e17881241b8396fa2f3: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/wals/e4d4a1bd80f44b668a2d504e47fb744a
I20240502 01:56:02.231983  5907 catalog_manager.cc:4830] TS 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245): tablet e4d4a1bd80f44b668a2d504e47fb744a (table rebalance_test_table_5 [id=c02dbeef0b7241e78a12e4bd7ef79e47]) successfully deleted
I20240502 01:56:02.242236 11601 ts_tablet_manager.cc:1875] T a398da35f18c4df394db82c5bf89aeb1 P a2c90ee440b44b708bccc2e2d5385cc0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:56:02.272639 11601 ts_tablet_manager.cc:1888] T a398da35f18c4df394db82c5bf89aeb1 P a2c90ee440b44b708bccc2e2d5385cc0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:56:02.273442 11601 log.cc:1196] T a398da35f18c4df394db82c5bf89aeb1 P a2c90ee440b44b708bccc2e2d5385cc0: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal/wals/a398da35f18c4df394db82c5bf89aeb1
I20240502 01:56:02.292248  5908 catalog_manager.cc:4830] TS a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197): tablet a398da35f18c4df394db82c5bf89aeb1 (table rebalance_test_table_5 [id=c02dbeef0b7241e78a12e4bd7ef79e47]) successfully deleted
I20240502 01:56:02.347473  6068 consensus_queue.cc:237] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:56:02.357731  8135 raft_consensus.cc:1261] T 899963a8260a42c4b73fb80eda2ecaa9 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:56:02.358781 11451 consensus_queue.cc:1035] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240502 01:56:02.368716  6335 raft_consensus.cc:1261] T 899963a8260a42c4b73fb80eda2ecaa9 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:56:02.369372 11451 consensus_queue.cc:1035] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:56:02.371204  6068 consensus_queue.cc:237] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:56:02.397682 11451 raft_consensus.cc:2932] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:02.405043  8406 raft_consensus.cc:1261] T c58d6b2401c047cd9af56a66c2e00105 P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:56:02.407032  6335 raft_consensus.cc:2932] T 899963a8260a42c4b73fb80eda2ecaa9 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:02.413635 11443 consensus_queue.cc:1035] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:56:02.416460  5917 catalog_manager.cc:5484] T 899963a8260a42c4b73fb80eda2ecaa9 P 94ffd7aaf85b4e17881241b8396fa2f3 reported cstate change: config changed from index 4 to 5, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New cstate: current_term: 1 leader_uuid: "218bc8c1f9ce46059898fc738179adf9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:02.421209  6447 tablet_service.cc:1514] Processing DeleteTablet for tablet 899963a8260a42c4b73fb80eda2ecaa9 with delete_type TABLET_DATA_TOMBSTONED (TS 07cdb213c77c4ffda1f999b66b725399 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:56564
W20240502 01:56:02.458361  6000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.131:44245, user_credentials={real_user=slave}} blocked reactor thread for 45367.7us
I20240502 01:56:02.459815  8135 raft_consensus.cc:1261] T c58d6b2401c047cd9af56a66c2e00105 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:56:02.460970  8135 raft_consensus.cc:2932] T 899963a8260a42c4b73fb80eda2ecaa9 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:02.468024  6469 raft_consensus.cc:1261] T c58d6b2401c047cd9af56a66c2e00105 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 01:56:02.471238  5999 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.136:44359, user_credentials={real_user=slave}} blocked reactor thread for 60205.3us
I20240502 01:56:02.473585 11675 consensus_queue.cc:1035] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:56:02.495671 11670 consensus_queue.cc:1035] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:56:02.498198 11674 tablet_replica.cc:330] T 899963a8260a42c4b73fb80eda2ecaa9 P 07cdb213c77c4ffda1f999b66b725399: stopping tablet replica
I20240502 01:56:02.515839 11443 raft_consensus.cc:2932] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:02.516103  8134 raft_consensus.cc:2932] T c58d6b2401c047cd9af56a66c2e00105 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:02.518703 11674 raft_consensus.cc:2229] T 899963a8260a42c4b73fb80eda2ecaa9 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:02.519049 11674 raft_consensus.cc:2258] T 899963a8260a42c4b73fb80eda2ecaa9 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:02.521538  6469 raft_consensus.cc:2932] T c58d6b2401c047cd9af56a66c2e00105 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:02.523540  5917 catalog_manager.cc:5484] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 reported cstate change: config changed from index 5 to 6, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New cstate: current_term: 1 leader_uuid: "218bc8c1f9ce46059898fc738179adf9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:02.531633  6180 tablet_service.cc:1514] Processing DeleteTablet for tablet c58d6b2401c047cd9af56a66c2e00105 with delete_type TABLET_DATA_TOMBSTONED (TS a2c90ee440b44b708bccc2e2d5385cc0 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41258
I20240502 01:56:02.532063 11601 tablet_replica.cc:330] T c58d6b2401c047cd9af56a66c2e00105 P a2c90ee440b44b708bccc2e2d5385cc0: stopping tablet replica
I20240502 01:56:02.532256 11601 raft_consensus.cc:2229] T c58d6b2401c047cd9af56a66c2e00105 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:02.532653 11601 raft_consensus.cc:2258] T c58d6b2401c047cd9af56a66c2e00105 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:02.533483 11601 ts_tablet_manager.cc:1875] T c58d6b2401c047cd9af56a66c2e00105 P a2c90ee440b44b708bccc2e2d5385cc0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:56:02.554035  8406 raft_consensus.cc:2932] T c58d6b2401c047cd9af56a66c2e00105 P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:02.562244 11601 ts_tablet_manager.cc:1888] T c58d6b2401c047cd9af56a66c2e00105 P a2c90ee440b44b708bccc2e2d5385cc0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:56:02.562525 11601 log.cc:1196] T c58d6b2401c047cd9af56a66c2e00105 P a2c90ee440b44b708bccc2e2d5385cc0: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal/wals/c58d6b2401c047cd9af56a66c2e00105
I20240502 01:56:02.563544  5908 catalog_manager.cc:4830] TS a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197): tablet c58d6b2401c047cd9af56a66c2e00105 (table rebalance_test_table_4 [id=1ed4a88787d6401481245c552170a456]) successfully deleted
I20240502 01:56:02.579856 11674 ts_tablet_manager.cc:1875] T 899963a8260a42c4b73fb80eda2ecaa9 P 07cdb213c77c4ffda1f999b66b725399: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:56:02.611337  6335 consensus_queue.cc:237] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:56:02.614768 11674 ts_tablet_manager.cc:1888] T 899963a8260a42c4b73fb80eda2ecaa9 P 07cdb213c77c4ffda1f999b66b725399: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:56:02.615088 11674 log.cc:1196] T 899963a8260a42c4b73fb80eda2ecaa9 P 07cdb213c77c4ffda1f999b66b725399: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal/wals/899963a8260a42c4b73fb80eda2ecaa9
I20240502 01:56:02.616178  5905 catalog_manager.cc:4830] TS 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867): tablet 899963a8260a42c4b73fb80eda2ecaa9 (table rebalance_test_table_6 [id=5606c3da105b4b86868c6c51ecdd39aa]) successfully deleted
I20240502 01:56:02.620544  8406 raft_consensus.cc:1261] T 3a7698f6bd4349ae832a22c853d76fea P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:56:02.621363 11182 consensus_queue.cc:1035] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:56:02.622052  8001 raft_consensus.cc:1261] T 3a7698f6bd4349ae832a22c853d76fea P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:56:02.623004  6204 raft_consensus.cc:1261] T 3a7698f6bd4349ae832a22c853d76fea P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:56:02.624666 11182 consensus_queue.cc:1035] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:56:02.625489 11182 consensus_queue.cc:1035] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 01:56:02.673434  6271 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.130:38197, user_credentials={real_user=slave}} blocked reactor thread for 48650.8us
I20240502 01:56:02.673820 11182 raft_consensus.cc:2932] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:02.676715  8001 raft_consensus.cc:2932] T 3a7698f6bd4349ae832a22c853d76fea P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:02.686782  8406 raft_consensus.cc:2932] T 3a7698f6bd4349ae832a22c853d76fea P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:02.688117  5916 catalog_manager.cc:5484] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 reported cstate change: config changed from index 5 to 6, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:02.697280  6048 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a7698f6bd4349ae832a22c853d76fea with delete_type TABLET_DATA_TOMBSTONED (TS 218bc8c1f9ce46059898fc738179adf9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50408
I20240502 01:56:02.701941  6204 raft_consensus.cc:2932] T 3a7698f6bd4349ae832a22c853d76fea P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:02.707396 11745 tablet_replica.cc:330] T 3a7698f6bd4349ae832a22c853d76fea P 218bc8c1f9ce46059898fc738179adf9: stopping tablet replica
I20240502 01:56:02.712916 11745 raft_consensus.cc:2229] T 3a7698f6bd4349ae832a22c853d76fea P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:02.713596 11745 raft_consensus.cc:2258] T 3a7698f6bd4349ae832a22c853d76fea P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:02.717283 11745 ts_tablet_manager.cc:1875] T 3a7698f6bd4349ae832a22c853d76fea P 218bc8c1f9ce46059898fc738179adf9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:56:02.735805 11745 ts_tablet_manager.cc:1888] T 3a7698f6bd4349ae832a22c853d76fea P 218bc8c1f9ce46059898fc738179adf9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:56:02.736788 11745 log.cc:1196] T 3a7698f6bd4349ae832a22c853d76fea P 218bc8c1f9ce46059898fc738179adf9: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal/wals/3a7698f6bd4349ae832a22c853d76fea
I20240502 01:56:02.744673  5907 catalog_manager.cc:4830] TS 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227): tablet 3a7698f6bd4349ae832a22c853d76fea (table rebalance_test_table_8 [id=fcde569cfdf747c2aaecac7d265b4b1f]) successfully deleted
I20240502 01:56:02.871060  6048 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:02.918247  6447 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:56:02.935403  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:56:02.956740  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:02.957206  8385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:03.016590  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:03.035158  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:03.053309  6048 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:03.089615  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:03.102898  8385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:03.113926  6447 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:56:03.138474  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:03.152195  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:56:03.175272  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:03.191156  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:03.197635  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:03.616341  6048 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:03.640166  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:03.647939  6447 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:56:03.670467  6447 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:56:03.672487  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:03.707379  8385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:56:03.722883  6402 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.134:42517, user_credentials={real_user=slave}} blocked reactor thread for 58941.8us
I20240502 01:56:03.761001  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:03.763765  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:03.786477  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:56:03.836184  6048 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:03.876843  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:03.893636  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:03.930639  8385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:03.968252  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:56:03.978580  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:04.017269  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:04.044278  6204 consensus_queue.cc:237] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:56:04.060894  6335 consensus_queue.cc:237] T 74eadf4c939c4c6383732cd74c5ec9d6 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 01:56:04.065371  8406 raft_consensus.cc:1261] T d7922fd756344d14a336ff0c54afdf73 P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Refusing update from remote peer a2c90ee440b44b708bccc2e2d5385cc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:56:04.067876 11397 consensus_queue.cc:1035] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:56:04.071981  6465 raft_consensus.cc:1261] T d7922fd756344d14a336ff0c54afdf73 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Refusing update from remote peer a2c90ee440b44b708bccc2e2d5385cc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:56:04.072655 11397 consensus_queue.cc:1035] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:56:04.087994  8001 raft_consensus.cc:1261] T 74eadf4c939c4c6383732cd74c5ec9d6 P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:56:04.089365 11729 consensus_queue.cc:1035] T 74eadf4c939c4c6383732cd74c5ec9d6 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:56:04.097759  6204 raft_consensus.cc:1261] T 74eadf4c939c4c6383732cd74c5ec9d6 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:56:04.098069 11921 raft_consensus.cc:2932] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:04.099638  6465 raft_consensus.cc:2932] T d7922fd756344d14a336ff0c54afdf73 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:04.105312 11729 consensus_queue.cc:1035] T 74eadf4c939c4c6383732cd74c5ec9d6 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:56:04.112498  5914 catalog_manager.cc:5484] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 reported cstate change: config changed from index 4 to 5, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New cstate: current_term: 1 leader_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:04.118814  8405 raft_consensus.cc:2932] T d7922fd756344d14a336ff0c54afdf73 P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:04.125068 11729 raft_consensus.cc:2932] T 74eadf4c939c4c6383732cd74c5ec9d6 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:56:04.125619  6200 raft_consensus.cc:2932] T 74eadf4c939c4c6383732cd74c5ec9d6 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:56:04.140165  8000 raft_consensus.cc:2932] T 74eadf4c939c4c6383732cd74c5ec9d6 P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:56:04.162494  5916 catalog_manager.cc:5484] T 74eadf4c939c4c6383732cd74c5ec9d6 P 60298ffea1dd4c25a38765c2ec9ec9d5 reported cstate change: config changed from index 4 to 5, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:56:04.187369  6316 tablet_service.cc:1514] Processing DeleteTablet for tablet d7922fd756344d14a336ff0c54afdf73 with delete_type TABLET_DATA_TOMBSTONED (TS 94ffd7aaf85b4e17881241b8396fa2f3 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41134
I20240502 01:56:04.195026 11956 tablet_replica.cc:330] T d7922fd756344d14a336ff0c54afdf73 P 94ffd7aaf85b4e17881241b8396fa2f3: stopping tablet replica
I20240502 01:56:04.195381 11956 raft_consensus.cc:2229] T d7922fd756344d14a336ff0c54afdf73 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:04.195756 11956 raft_consensus.cc:2258] T d7922fd756344d14a336ff0c54afdf73 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:04.203416 11956 ts_tablet_manager.cc:1875] T d7922fd756344d14a336ff0c54afdf73 P 94ffd7aaf85b4e17881241b8396fa2f3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:56:04.222756  6048 tablet_service.cc:1514] Processing DeleteTablet for tablet 74eadf4c939c4c6383732cd74c5ec9d6 with delete_type TABLET_DATA_TOMBSTONED (TS 218bc8c1f9ce46059898fc738179adf9 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:50408
I20240502 01:56:04.223775 11956 ts_tablet_manager.cc:1888] T d7922fd756344d14a336ff0c54afdf73 P 94ffd7aaf85b4e17881241b8396fa2f3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:56:04.224026 11956 log.cc:1196] T d7922fd756344d14a336ff0c54afdf73 P 94ffd7aaf85b4e17881241b8396fa2f3: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/wals/d7922fd756344d14a336ff0c54afdf73
I20240502 01:56:04.224972  5907 catalog_manager.cc:4830] TS 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245): tablet d7922fd756344d14a336ff0c54afdf73 (table rebalance_test_table_5 [id=c02dbeef0b7241e78a12e4bd7ef79e47]) successfully deleted
I20240502 01:56:04.225381 11968 tablet_replica.cc:330] T 74eadf4c939c4c6383732cd74c5ec9d6 P 218bc8c1f9ce46059898fc738179adf9: stopping tablet replica
I20240502 01:56:04.225679 11968 raft_consensus.cc:2229] T 74eadf4c939c4c6383732cd74c5ec9d6 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:04.226135 11968 raft_consensus.cc:2258] T 74eadf4c939c4c6383732cd74c5ec9d6 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:04.227016 11968 ts_tablet_manager.cc:1875] T 74eadf4c939c4c6383732cd74c5ec9d6 P 218bc8c1f9ce46059898fc738179adf9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:56:04.248888 11968 ts_tablet_manager.cc:1888] T 74eadf4c939c4c6383732cd74c5ec9d6 P 218bc8c1f9ce46059898fc738179adf9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:56:04.249171 11968 log.cc:1196] T 74eadf4c939c4c6383732cd74c5ec9d6 P 218bc8c1f9ce46059898fc738179adf9: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal/wals/74eadf4c939c4c6383732cd74c5ec9d6
I20240502 01:56:04.250531  5907 catalog_manager.cc:4830] TS 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227): tablet 74eadf4c939c4c6383732cd74c5ec9d6 (table rebalance_test_table_4 [id=1ed4a88787d6401481245c552170a456]) successfully deleted
I20240502 01:56:04.392405  6335 consensus_queue.cc:237] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:56:04.410859  6068 consensus_queue.cc:237] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:56:04.432879  8405 raft_consensus.cc:1261] T d8a6db6ae11e4cb8881aee95bdb0e59d P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:56:04.433509 11670 consensus_queue.cc:1035] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:56:04.443117  6336 raft_consensus.cc:1261] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:56:04.445219 11670 consensus_queue.cc:1035] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:56:04.465256 11670 raft_consensus.cc:2932] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:04.469074  8406 raft_consensus.cc:2932] T d8a6db6ae11e4cb8881aee95bdb0e59d P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:04.475126  5920 catalog_manager.cc:5484] T d8a6db6ae11e4cb8881aee95bdb0e59d P 6d34e50fb5dd4a1db58013d4863b8992 reported cstate change: config changed from index 4 to 5, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New cstate: current_term: 1 leader_uuid: "218bc8c1f9ce46059898fc738179adf9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:04.479969  6447 tablet_service.cc:1514] Processing DeleteTablet for tablet d8a6db6ae11e4cb8881aee95bdb0e59d with delete_type TABLET_DATA_TOMBSTONED (TS 07cdb213c77c4ffda1f999b66b725399 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:56564
I20240502 01:56:04.487641  8134 raft_consensus.cc:1261] T 7f2519b0c6404799ad84d120b45a7ea4 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:56:04.506183 11934 consensus_queue.cc:1035] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.010s
I20240502 01:56:04.511049  6200 raft_consensus.cc:1261] T 7f2519b0c6404799ad84d120b45a7ea4 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:56:04.511883 11935 consensus_queue.cc:1035] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:56:04.536767  8385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:04.541692  6336 raft_consensus.cc:2932] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:04.564543  6048 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:04.577569  6446 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:56:04.595626 11935 raft_consensus.cc:2932] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:04.601843  8135 raft_consensus.cc:2932] T 7f2519b0c6404799ad84d120b45a7ea4 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:04.628669  5916 catalog_manager.cc:5484] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 reported cstate change: config changed from index 4 to 5, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:04.633360  6048 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f2519b0c6404799ad84d120b45a7ea4 with delete_type TABLET_DATA_TOMBSTONED (TS 218bc8c1f9ce46059898fc738179adf9 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:50408
I20240502 01:56:04.633767 11968 tablet_replica.cc:330] T 7f2519b0c6404799ad84d120b45a7ea4 P 218bc8c1f9ce46059898fc738179adf9: stopping tablet replica
I20240502 01:56:04.634002 11968 raft_consensus.cc:2229] T 7f2519b0c6404799ad84d120b45a7ea4 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:04.634230 11968 raft_consensus.cc:2258] T 7f2519b0c6404799ad84d120b45a7ea4 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:04.634977 11968 ts_tablet_manager.cc:1875] T 7f2519b0c6404799ad84d120b45a7ea4 P 218bc8c1f9ce46059898fc738179adf9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:56:04.641680 11968 ts_tablet_manager.cc:1888] T 7f2519b0c6404799ad84d120b45a7ea4 P 218bc8c1f9ce46059898fc738179adf9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:56:04.641906 11968 log.cc:1196] T 7f2519b0c6404799ad84d120b45a7ea4 P 218bc8c1f9ce46059898fc738179adf9: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal/wals/7f2519b0c6404799ad84d120b45a7ea4
I20240502 01:56:04.642750  5907 catalog_manager.cc:4830] TS 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227): tablet 7f2519b0c6404799ad84d120b45a7ea4 (table rebalance_test_table_3 [id=63a163be603345428cc84c86b96467f0]) successfully deleted
I20240502 01:56:04.673272  6204 raft_consensus.cc:2932] T 7f2519b0c6404799ad84d120b45a7ea4 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:04.685853 12022 tablet_replica.cc:330] T d8a6db6ae11e4cb8881aee95bdb0e59d P 07cdb213c77c4ffda1f999b66b725399: stopping tablet replica
I20240502 01:56:04.692194 12022 raft_consensus.cc:2229] T d8a6db6ae11e4cb8881aee95bdb0e59d P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:04.694711 12022 raft_consensus.cc:2258] T d8a6db6ae11e4cb8881aee95bdb0e59d P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:04.697932 12022 ts_tablet_manager.cc:1875] T d8a6db6ae11e4cb8881aee95bdb0e59d P 07cdb213c77c4ffda1f999b66b725399: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:56:04.720887  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:56:04.728034  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:04.731171 12022 ts_tablet_manager.cc:1888] T d8a6db6ae11e4cb8881aee95bdb0e59d P 07cdb213c77c4ffda1f999b66b725399: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:56:04.731406 12022 log.cc:1196] T d8a6db6ae11e4cb8881aee95bdb0e59d P 07cdb213c77c4ffda1f999b66b725399: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal/wals/d8a6db6ae11e4cb8881aee95bdb0e59d
I20240502 01:56:04.734877  5905 catalog_manager.cc:4830] TS 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867): tablet d8a6db6ae11e4cb8881aee95bdb0e59d (table rebalance_test_table_8 [id=fcde569cfdf747c2aaecac7d265b4b1f]) successfully deleted
I20240502 01:56:04.743831  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:04.744789  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:04.749310  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:05.138522  6447 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:56:05.151594  6048 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:05.175706  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:05.215193  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:05.266296  6200 consensus_queue.cc:237] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:56:05.274367  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:05.286325  8135 raft_consensus.cc:1261] T ca458907378e42d48adaa0ee647db4cd P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Refusing update from remote peer a2c90ee440b44b708bccc2e2d5385cc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:56:05.287231 12036 consensus_queue.cc:1035] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:56:05.289017  6068 raft_consensus.cc:1261] T ca458907378e42d48adaa0ee647db4cd P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Refusing update from remote peer a2c90ee440b44b708bccc2e2d5385cc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:56:05.290956 12036 consensus_queue.cc:1035] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:56:05.312919  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:05.333148  8385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:05.340718  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:56:05.372799  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:05.422876  8405 raft_consensus.cc:1261] T ca458907378e42d48adaa0ee647db4cd P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Refusing update from remote peer a2c90ee440b44b708bccc2e2d5385cc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:56:05.427619 12147 raft_consensus.cc:2932] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:05.430572  6048 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:05.431253  8385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:05.432600  6067 raft_consensus.cc:2932] T ca458907378e42d48adaa0ee647db4cd P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:05.441287  5916 catalog_manager.cc:5484] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 reported cstate change: config changed from index 5 to 6, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New cstate: current_term: 1 leader_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:05.445848  6447 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:56:05.463017  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:05.466277 12036 consensus_queue.cc:1035] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:56:05.470005  8135 raft_consensus.cc:2932] T ca458907378e42d48adaa0ee647db4cd P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:05.477456  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:05.485818  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:56:05.537027  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:05.564033  8405 raft_consensus.cc:2932] T ca458907378e42d48adaa0ee647db4cd P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:05.592430  6447 tablet_service.cc:1514] Processing DeleteTablet for tablet ca458907378e42d48adaa0ee647db4cd with delete_type TABLET_DATA_TOMBSTONED (TS 07cdb213c77c4ffda1f999b66b725399 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56564
I20240502 01:56:05.605320 12184 tablet_replica.cc:330] T ca458907378e42d48adaa0ee647db4cd P 07cdb213c77c4ffda1f999b66b725399: stopping tablet replica
I20240502 01:56:05.606717 12184 raft_consensus.cc:2229] T ca458907378e42d48adaa0ee647db4cd P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:05.607560 12184 raft_consensus.cc:2258] T ca458907378e42d48adaa0ee647db4cd P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:05.608502 12184 ts_tablet_manager.cc:1875] T ca458907378e42d48adaa0ee647db4cd P 07cdb213c77c4ffda1f999b66b725399: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:56:05.677343 12184 ts_tablet_manager.cc:1888] T ca458907378e42d48adaa0ee647db4cd P 07cdb213c77c4ffda1f999b66b725399: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:56:05.677548 12184 log.cc:1196] T ca458907378e42d48adaa0ee647db4cd P 07cdb213c77c4ffda1f999b66b725399: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal/wals/ca458907378e42d48adaa0ee647db4cd
I20240502 01:56:05.678448  5905 catalog_manager.cc:4830] TS 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867): tablet ca458907378e42d48adaa0ee647db4cd (table rebalance_test_table_2 [id=ef7ebedfbbff4054b074bf7a7622b632]) successfully deleted
I20240502 01:56:05.831667  6447 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:56:05.844079  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:05.878072  6048 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:05.885136  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:05.918066  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:05.919530  8385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:05.952636  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:05.956849  6447 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:56:05.987025  6048 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:05.992446  8385 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:06.069921  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:56:06.073137  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:56:06.128170  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:06.194100  6465 consensus_queue.cc:237] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 01:56:06.196985  8000 raft_consensus.cc:1261] T 4486b413e5c14ec6953e4895ee5797cf P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:56:06.197789 11874 consensus_queue.cc:1035] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:56:06.201387  6200 raft_consensus.cc:1261] T 4486b413e5c14ec6953e4895ee5797cf P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:56:06.202320 11874 consensus_queue.cc:1035] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:56:06.256682 11874 raft_consensus.cc:2932] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:56:06.260699  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:06.261972  6204 raft_consensus.cc:2932] T 4486b413e5c14ec6953e4895ee5797cf P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:56:06.271492  5912 catalog_manager.cc:5484] T 4486b413e5c14ec6953e4895ee5797cf P a2c90ee440b44b708bccc2e2d5385cc0 reported cstate change: config changed from index 4 to 5, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New cstate: current_term: 1 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:56:06.297345  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:06.314968  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:56:06.339401  6403 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.130:38197, user_credentials={real_user=slave}} blocked reactor thread for 76483.5us
W20240502 01:56:06.351809  6402 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.134:42517, user_credentials={real_user=slave}} blocked reactor thread for 89228.8us
I20240502 01:56:06.352895  8000 raft_consensus.cc:2932] T 4486b413e5c14ec6953e4895ee5797cf P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:56:06.371587  6336 consensus_queue.cc:237] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:56:06.383213  6048 tablet_service.cc:1514] Processing DeleteTablet for tablet 4486b413e5c14ec6953e4895ee5797cf with delete_type TABLET_DATA_TOMBSTONED (TS 218bc8c1f9ce46059898fc738179adf9 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:50408
I20240502 01:56:06.398452 12295 tablet_replica.cc:330] T 4486b413e5c14ec6953e4895ee5797cf P 218bc8c1f9ce46059898fc738179adf9: stopping tablet replica
I20240502 01:56:06.399777  8269 raft_consensus.cc:1261] T 866f87ea9f7f48edb667b1b2eb3fa239 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:56:06.401028 12295 raft_consensus.cc:2229] T 4486b413e5c14ec6953e4895ee5797cf P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:06.402279 11935 consensus_queue.cc:1035] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:56:06.402925 12295 raft_consensus.cc:2258] T 4486b413e5c14ec6953e4895ee5797cf P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:06.404680  6465 raft_consensus.cc:1261] T 866f87ea9f7f48edb667b1b2eb3fa239 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:56:06.405679 11725 consensus_queue.cc:1035] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:56:06.415108 12295 ts_tablet_manager.cc:1875] T 4486b413e5c14ec6953e4895ee5797cf P 218bc8c1f9ce46059898fc738179adf9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:56:06.422647 12295 ts_tablet_manager.cc:1888] T 4486b413e5c14ec6953e4895ee5797cf P 218bc8c1f9ce46059898fc738179adf9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:56:06.422833 12295 log.cc:1196] T 4486b413e5c14ec6953e4895ee5797cf P 218bc8c1f9ce46059898fc738179adf9: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal/wals/4486b413e5c14ec6953e4895ee5797cf
I20240502 01:56:06.424525  5907 catalog_manager.cc:4830] TS 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227): tablet 4486b413e5c14ec6953e4895ee5797cf (table rebalance_test_table_6 [id=5606c3da105b4b86868c6c51ecdd39aa]) successfully deleted
I20240502 01:56:06.465183 11725 raft_consensus.cc:2932] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:06.470942  6465 raft_consensus.cc:2932] T 866f87ea9f7f48edb667b1b2eb3fa239 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:06.479318  8269 raft_consensus.cc:2932] T 866f87ea9f7f48edb667b1b2eb3fa239 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:06.481545  5912 catalog_manager.cc:5484] T 866f87ea9f7f48edb667b1b2eb3fa239 P 07cdb213c77c4ffda1f999b66b725399 reported cstate change: config changed from index 4 to 5, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:06.486223  6180 tablet_service.cc:1514] Processing DeleteTablet for tablet 866f87ea9f7f48edb667b1b2eb3fa239 with delete_type TABLET_DATA_TOMBSTONED (TS a2c90ee440b44b708bccc2e2d5385cc0 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41258
I20240502 01:56:06.512898 12309 tablet_replica.cc:330] T 866f87ea9f7f48edb667b1b2eb3fa239 P a2c90ee440b44b708bccc2e2d5385cc0: stopping tablet replica
I20240502 01:56:06.527395 12309 raft_consensus.cc:2229] T 866f87ea9f7f48edb667b1b2eb3fa239 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:06.528486 12309 raft_consensus.cc:2258] T 866f87ea9f7f48edb667b1b2eb3fa239 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:06.531412 12309 ts_tablet_manager.cc:1875] T 866f87ea9f7f48edb667b1b2eb3fa239 P a2c90ee440b44b708bccc2e2d5385cc0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:56:06.549015 12309 ts_tablet_manager.cc:1888] T 866f87ea9f7f48edb667b1b2eb3fa239 P a2c90ee440b44b708bccc2e2d5385cc0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:56:06.549180 12309 log.cc:1196] T 866f87ea9f7f48edb667b1b2eb3fa239 P a2c90ee440b44b708bccc2e2d5385cc0: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal/wals/866f87ea9f7f48edb667b1b2eb3fa239
I20240502 01:56:06.550163  5908 catalog_manager.cc:4830] TS a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197): tablet 866f87ea9f7f48edb667b1b2eb3fa239 (table rebalance_test_table_3 [id=63a163be603345428cc84c86b96467f0]) successfully deleted
I20240502 01:56:06.567623  6465 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "265a9645e1f141cb88c786b9aef8f90e"
dest_uuid: "07cdb213c77c4ffda1f999b66b725399"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:44946
I20240502 01:56:06.568061  6465 raft_consensus.cc:595] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:56:06.728459  6447 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:56:06.728861 12275 raft_consensus.cc:979] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399: : Instructing follower 6d34e50fb5dd4a1db58013d4863b8992 to start an election
I20240502 01:56:06.729028 12275 raft_consensus.cc:1067] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Signalling peer 6d34e50fb5dd4a1db58013d4863b8992 to start an election
I20240502 01:56:06.730233  8405 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "265a9645e1f141cb88c786b9aef8f90e"
dest_uuid: "6d34e50fb5dd4a1db58013d4863b8992"
 from {username='slave'} at 127.25.109.132:41331
I20240502 01:56:06.730558  8405 raft_consensus.cc:482] T 265a9645e1f141cb88c786b9aef8f90e P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:56:06.730728  8405 raft_consensus.cc:3037] T 265a9645e1f141cb88c786b9aef8f90e P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:56:06.733857  8405 raft_consensus.cc:504] T 265a9645e1f141cb88c786b9aef8f90e P 6d34e50fb5dd4a1db58013d4863b8992 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:56:06.736459  6465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "265a9645e1f141cb88c786b9aef8f90e" candidate_uuid: "6d34e50fb5dd4a1db58013d4863b8992" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "07cdb213c77c4ffda1f999b66b725399"
I20240502 01:56:06.736737  6465 raft_consensus.cc:3032] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:56:06.736828  6465 raft_consensus.cc:729] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 07cdb213c77c4ffda1f999b66b725399, State: Running, Role: LEADER
I20240502 01:56:06.736754  6336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "265a9645e1f141cb88c786b9aef8f90e" candidate_uuid: "6d34e50fb5dd4a1db58013d4863b8992" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "94ffd7aaf85b4e17881241b8396fa2f3"
I20240502 01:56:06.736986  6336 raft_consensus.cc:3037] T 265a9645e1f141cb88c786b9aef8f90e P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:56:06.737233  6465 consensus_queue.cc:260] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:56:06.737802  6465 raft_consensus.cc:3037] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:56:06.739840  6336 raft_consensus.cc:2454] T 265a9645e1f141cb88c786b9aef8f90e P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d34e50fb5dd4a1db58013d4863b8992 in term 2.
I20240502 01:56:06.741466  6067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "265a9645e1f141cb88c786b9aef8f90e" candidate_uuid: "6d34e50fb5dd4a1db58013d4863b8992" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "218bc8c1f9ce46059898fc738179adf9"
I20240502 01:56:06.741732  6067 raft_consensus.cc:3037] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:56:06.743062  6465 raft_consensus.cc:2454] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d34e50fb5dd4a1db58013d4863b8992 in term 2.
I20240502 01:56:06.743746  8335 leader_election.cc:302] T 265a9645e1f141cb88c786b9aef8f90e P 6d34e50fb5dd4a1db58013d4863b8992 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 07cdb213c77c4ffda1f999b66b725399, 6d34e50fb5dd4a1db58013d4863b8992, 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 
I20240502 01:56:06.744695  8405 leader_election.cc:288] T 265a9645e1f141cb88c786b9aef8f90e P 6d34e50fb5dd4a1db58013d4863b8992 [CANDIDATE]: Term 2 election: Requested vote from peers 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867), 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245), 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
I20240502 01:56:06.744704  6067 raft_consensus.cc:2454] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d34e50fb5dd4a1db58013d4863b8992 in term 2.
I20240502 01:56:06.754644 12377 raft_consensus.cc:2781] T 265a9645e1f141cb88c786b9aef8f90e P 6d34e50fb5dd4a1db58013d4863b8992 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:56:06.763144 12377 raft_consensus.cc:686] T 265a9645e1f141cb88c786b9aef8f90e P 6d34e50fb5dd4a1db58013d4863b8992 [term 2 LEADER]: Becoming Leader. State: Replica: 6d34e50fb5dd4a1db58013d4863b8992, State: Running, Role: LEADER
I20240502 01:56:06.764506 12377 consensus_queue.cc:237] T 265a9645e1f141cb88c786b9aef8f90e P 6d34e50fb5dd4a1db58013d4863b8992 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:56:06.769397  8135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "265a9645e1f141cb88c786b9aef8f90e" candidate_uuid: "6d34e50fb5dd4a1db58013d4863b8992" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "9db17f84d37944098cbeb9e0d8456efb"
I20240502 01:56:06.769824  8135 raft_consensus.cc:3037] T 265a9645e1f141cb88c786b9aef8f90e P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:56:06.779206  6048 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:06.780733  8135 raft_consensus.cc:2454] T 265a9645e1f141cb88c786b9aef8f90e P 9db17f84d37944098cbeb9e0d8456efb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d34e50fb5dd4a1db58013d4863b8992 in term 2.
I20240502 01:56:06.793762  6067 consensus_queue.cc:237] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:56:06.797250  5922 catalog_manager.cc:5484] T 265a9645e1f141cb88c786b9aef8f90e P 6d34e50fb5dd4a1db58013d4863b8992 reported cstate change: term changed from 1 to 2, leader changed from 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) to 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136). New cstate: current_term: 2 leader_uuid: "6d34e50fb5dd4a1db58013d4863b8992" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:06.810129  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:06.815691  8135 raft_consensus.cc:1261] T 4f4088f3523f4cd0afdf0de0f447c7af P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:56:06.816362 12256 consensus_queue.cc:1035] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:56:06.818240  8405 raft_consensus.cc:1261] T 4f4088f3523f4cd0afdf0de0f447c7af P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:56:06.819150  6200 raft_consensus.cc:1261] T 4f4088f3523f4cd0afdf0de0f447c7af P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:56:06.819737 12256 consensus_queue.cc:1035] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:56:06.821799 12256 consensus_queue.cc:1035] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:56:06.893801  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:06.917937  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:06.920073  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:56:06.934242 12407 raft_consensus.cc:2932] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:06.957278  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:06.957983  8134 raft_consensus.cc:2932] T 4f4088f3523f4cd0afdf0de0f447c7af P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:06.964537  8405 raft_consensus.cc:2932] T 4f4088f3523f4cd0afdf0de0f447c7af P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:06.967402  8385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:06.973697  6200 raft_consensus.cc:2932] T 4f4088f3523f4cd0afdf0de0f447c7af P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:06.974319  5922 catalog_manager.cc:5484] T 4f4088f3523f4cd0afdf0de0f447c7af P 6d34e50fb5dd4a1db58013d4863b8992 reported cstate change: config changed from index 5 to 6, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New cstate: current_term: 1 leader_uuid: "218bc8c1f9ce46059898fc738179adf9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:06.979326  6316 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f4088f3523f4cd0afdf0de0f447c7af with delete_type TABLET_DATA_TOMBSTONED (TS 94ffd7aaf85b4e17881241b8396fa2f3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41134
I20240502 01:56:07.029512 12416 tablet_replica.cc:330] T 4f4088f3523f4cd0afdf0de0f447c7af P 94ffd7aaf85b4e17881241b8396fa2f3: stopping tablet replica
I20240502 01:56:07.036417 12416 raft_consensus.cc:2229] T 4f4088f3523f4cd0afdf0de0f447c7af P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:07.037147 12416 raft_consensus.cc:2258] T 4f4088f3523f4cd0afdf0de0f447c7af P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:07.039949 12416 ts_tablet_manager.cc:1875] T 4f4088f3523f4cd0afdf0de0f447c7af P 94ffd7aaf85b4e17881241b8396fa2f3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 01:56:07.094614  6268 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.135:40543, user_credentials={real_user=slave}} blocked reactor thread for 67140.7us
I20240502 01:56:07.102636 12416 ts_tablet_manager.cc:1888] T 4f4088f3523f4cd0afdf0de0f447c7af P 94ffd7aaf85b4e17881241b8396fa2f3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:56:07.121769 12416 log.cc:1196] T 4f4088f3523f4cd0afdf0de0f447c7af P 94ffd7aaf85b4e17881241b8396fa2f3: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/wals/4f4088f3523f4cd0afdf0de0f447c7af
I20240502 01:56:07.134004  5907 catalog_manager.cc:4830] TS 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245): tablet 4f4088f3523f4cd0afdf0de0f447c7af (table rebalance_test_table_7 [id=c2701645b8bc497aaf63e970157bdd80]) successfully deleted
I20240502 01:56:07.154002  6336 raft_consensus.cc:1261] T 265a9645e1f141cb88c786b9aef8f90e P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Refusing update from remote peer 6d34e50fb5dd4a1db58013d4863b8992: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:56:07.154949 12377 consensus_queue.cc:1035] T 265a9645e1f141cb88c786b9aef8f90e P 6d34e50fb5dd4a1db58013d4863b8992 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:56:07.272308  6465 raft_consensus.cc:1261] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399 [term 2 FOLLOWER]: Refusing update from remote peer 6d34e50fb5dd4a1db58013d4863b8992: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:56:07.273052 12377 consensus_queue.cc:1035] T 265a9645e1f141cb88c786b9aef8f90e P 6d34e50fb5dd4a1db58013d4863b8992 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:56:07.304541  6067 raft_consensus.cc:1261] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Refusing update from remote peer 6d34e50fb5dd4a1db58013d4863b8992: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:56:07.305312 12377 consensus_queue.cc:1035] T 265a9645e1f141cb88c786b9aef8f90e P 6d34e50fb5dd4a1db58013d4863b8992 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:56:07.318722  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:07.330662  6048 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:07.339258  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:07.371079  8134 raft_consensus.cc:1261] T 265a9645e1f141cb88c786b9aef8f90e P 9db17f84d37944098cbeb9e0d8456efb [term 2 FOLLOWER]: Refusing update from remote peer 6d34e50fb5dd4a1db58013d4863b8992: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:56:07.372732 12475 consensus_queue.cc:1035] T 265a9645e1f141cb88c786b9aef8f90e P 6d34e50fb5dd4a1db58013d4863b8992 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:56:07.387305  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:07.423110  8385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:07.489929  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:56:07.509660  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:07.555228  6336 consensus_queue.cc:237] T ac3325a06d8841718158e1f5f181a398 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 01:56:07.570173  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:07.591171  8000 raft_consensus.cc:1261] T ac3325a06d8841718158e1f5f181a398 P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:56:07.591945 12422 consensus_queue.cc:1035] T ac3325a06d8841718158e1f5f181a398 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:56:07.611934 12422 raft_consensus.cc:2932] T ac3325a06d8841718158e1f5f181a398 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:56:07.616919  8001 raft_consensus.cc:2932] T ac3325a06d8841718158e1f5f181a398 P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:56:07.622571  5922 catalog_manager.cc:5484] T ac3325a06d8841718158e1f5f181a398 P 94ffd7aaf85b4e17881241b8396fa2f3 reported cstate change: config changed from index 4 to 5, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:56:07.638006  6465 raft_consensus.cc:1261] T ac3325a06d8841718158e1f5f181a398 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:56:07.639165 12424 consensus_queue.cc:1035] T ac3325a06d8841718158e1f5f181a398 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:56:07.678050  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:07.690987  6180 tablet_service.cc:1514] Processing DeleteTablet for tablet ac3325a06d8841718158e1f5f181a398 with delete_type TABLET_DATA_TOMBSTONED (TS a2c90ee440b44b708bccc2e2d5385cc0 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41258
I20240502 01:56:07.725258  8385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:07.725430  6465 raft_consensus.cc:2932] T ac3325a06d8841718158e1f5f181a398 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:56:07.741722  6048 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:07.748646  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:56:07.761700 12555 tablet_replica.cc:330] T ac3325a06d8841718158e1f5f181a398 P a2c90ee440b44b708bccc2e2d5385cc0: stopping tablet replica
I20240502 01:56:07.765053 12555 raft_consensus.cc:2229] T ac3325a06d8841718158e1f5f181a398 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:07.770290 12555 raft_consensus.cc:2258] T ac3325a06d8841718158e1f5f181a398 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:07.770913  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:07.772679  6048 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:07.801535  8385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:07.811511 12555 ts_tablet_manager.cc:1875] T ac3325a06d8841718158e1f5f181a398 P a2c90ee440b44b708bccc2e2d5385cc0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:56:07.814638  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:07.871325 12555 ts_tablet_manager.cc:1888] T ac3325a06d8841718158e1f5f181a398 P a2c90ee440b44b708bccc2e2d5385cc0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:56:07.871521 12555 log.cc:1196] T ac3325a06d8841718158e1f5f181a398 P a2c90ee440b44b708bccc2e2d5385cc0: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal/wals/ac3325a06d8841718158e1f5f181a398
I20240502 01:56:07.872515  5908 catalog_manager.cc:4830] TS a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197): tablet ac3325a06d8841718158e1f5f181a398 (table rebalance_test_table_1 [id=0053d72412224a79be83b7e007b51934]) successfully deleted
I20240502 01:56:07.881296  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:07.887109  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:07.899058  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:07.913066  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:07.920101  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:07.926481  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:07.985102  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:08.047176  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:56:08.378818  6067 consensus_queue.cc:237] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:56:08.384007  8134 raft_consensus.cc:1261] T 4f4088f3523f4cd0afdf0de0f447c7af P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:56:08.384825 12409 consensus_queue.cc:1035] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:56:08.385365  8406 consensus_queue.cc:237] T 265a9645e1f141cb88c786b9aef8f90e P 6d34e50fb5dd4a1db58013d4863b8992 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:56:08.387008  8405 raft_consensus.cc:1261] T 4f4088f3523f4cd0afdf0de0f447c7af P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:56:08.389107 12409 consensus_queue.cc:1035] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:56:08.389252  6068 raft_consensus.cc:1261] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Refusing update from remote peer 6d34e50fb5dd4a1db58013d4863b8992: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:56:08.389238  8135 raft_consensus.cc:1261] T 265a9645e1f141cb88c786b9aef8f90e P 9db17f84d37944098cbeb9e0d8456efb [term 2 FOLLOWER]: Refusing update from remote peer 6d34e50fb5dd4a1db58013d4863b8992: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:56:08.390110 12477 consensus_queue.cc:1035] T 265a9645e1f141cb88c786b9aef8f90e P 6d34e50fb5dd4a1db58013d4863b8992 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:56:08.390682 12477 consensus_queue.cc:1035] T 265a9645e1f141cb88c786b9aef8f90e P 6d34e50fb5dd4a1db58013d4863b8992 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:56:08.394744 12477 raft_consensus.cc:2932] T 265a9645e1f141cb88c786b9aef8f90e P 6d34e50fb5dd4a1db58013d4863b8992 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:08.398226  6068 raft_consensus.cc:2932] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:08.405849 12409 raft_consensus.cc:2932] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:08.414988  8135 raft_consensus.cc:2932] T 4f4088f3523f4cd0afdf0de0f447c7af P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:08.453316  6334 consensus_queue.cc:237] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:56:08.454953  8405 raft_consensus.cc:2932] T 4f4088f3523f4cd0afdf0de0f447c7af P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:08.475355  8134 raft_consensus.cc:2932] T 265a9645e1f141cb88c786b9aef8f90e P 9db17f84d37944098cbeb9e0d8456efb [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:08.481002  6336 raft_consensus.cc:1261] T 265a9645e1f141cb88c786b9aef8f90e P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Refusing update from remote peer 6d34e50fb5dd4a1db58013d4863b8992: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:56:08.481911 12671 consensus_queue.cc:1035] T 265a9645e1f141cb88c786b9aef8f90e P 6d34e50fb5dd4a1db58013d4863b8992 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:56:08.486872  5921 catalog_manager.cc:5484] T 265a9645e1f141cb88c786b9aef8f90e P 6d34e50fb5dd4a1db58013d4863b8992 reported cstate change: config changed from index 5 to 7, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New cstate: current_term: 2 leader_uuid: "6d34e50fb5dd4a1db58013d4863b8992" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:08.489435  5921 catalog_manager.cc:5484] T 4f4088f3523f4cd0afdf0de0f447c7af P 6d34e50fb5dd4a1db58013d4863b8992 reported cstate change: config changed from index 6 to 7, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New cstate: current_term: 1 leader_uuid: "218bc8c1f9ce46059898fc738179adf9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:08.492563  6336 raft_consensus.cc:2932] T 265a9645e1f141cb88c786b9aef8f90e P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
W20240502 01:56:08.515616  5999 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.136:44359, user_credentials={real_user=slave}} blocked reactor thread for 51872.2us
I20240502 01:56:08.524657  8134 raft_consensus.cc:1261] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:56:08.524871  8269 raft_consensus.cc:1261] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:56:08.525480 12424 consensus_queue.cc:1035] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:56:08.526189 12422 consensus_queue.cc:1035] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:56:08.529784  6068 raft_consensus.cc:1261] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:56:08.530532 12422 consensus_queue.cc:1035] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:56:08.566161  6447 tablet_service.cc:1514] Processing DeleteTablet for tablet 265a9645e1f141cb88c786b9aef8f90e with delete_type TABLET_DATA_TOMBSTONED (TS 07cdb213c77c4ffda1f999b66b725399 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:56564
I20240502 01:56:08.566812  6180 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f4088f3523f4cd0afdf0de0f447c7af with delete_type TABLET_DATA_TOMBSTONED (TS a2c90ee440b44b708bccc2e2d5385cc0 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:41258
I20240502 01:56:08.617745 12424 raft_consensus.cc:2932] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:08.624233  8135 raft_consensus.cc:2932] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:08.638495 12700 tablet_replica.cc:330] T 4f4088f3523f4cd0afdf0de0f447c7af P a2c90ee440b44b708bccc2e2d5385cc0: stopping tablet replica
I20240502 01:56:08.638835 12700 raft_consensus.cc:2229] T 4f4088f3523f4cd0afdf0de0f447c7af P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:08.639171 12700 raft_consensus.cc:2258] T 4f4088f3523f4cd0afdf0de0f447c7af P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:08.640092 12700 ts_tablet_manager.cc:1875] T 4f4088f3523f4cd0afdf0de0f447c7af P a2c90ee440b44b708bccc2e2d5385cc0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:56:08.640070  5916 catalog_manager.cc:5484] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 reported cstate change: config changed from index 5 to 6, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:08.643035 12698 tablet_replica.cc:330] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399: stopping tablet replica
I20240502 01:56:08.643327 12698 raft_consensus.cc:2229] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:08.643684 12698 raft_consensus.cc:2258] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:08.644515 12698 ts_tablet_manager.cc:1875] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:56:08.645262  6446 tablet_service.cc:1514] Processing DeleteTablet for tablet afa5561f8c0e4c0fa9bbbd5da61348c7 with delete_type TABLET_DATA_TOMBSTONED (TS 07cdb213c77c4ffda1f999b66b725399 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56564
I20240502 01:56:08.654297  6068 raft_consensus.cc:2932] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:08.665963  8269 raft_consensus.cc:2932] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:08.676904 12698 ts_tablet_manager.cc:1888] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 01:56:08.694231 12700 ts_tablet_manager.cc:1888] T 4f4088f3523f4cd0afdf0de0f447c7af P a2c90ee440b44b708bccc2e2d5385cc0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:56:08.694809 12700 log.cc:1196] T 4f4088f3523f4cd0afdf0de0f447c7af P a2c90ee440b44b708bccc2e2d5385cc0: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal/wals/4f4088f3523f4cd0afdf0de0f447c7af
I20240502 01:56:08.695163 12698 log.cc:1196] T 265a9645e1f141cb88c786b9aef8f90e P 07cdb213c77c4ffda1f999b66b725399: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal/wals/265a9645e1f141cb88c786b9aef8f90e
I20240502 01:56:08.696241  5908 catalog_manager.cc:4830] TS a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197): tablet 4f4088f3523f4cd0afdf0de0f447c7af (table rebalance_test_table_7 [id=c2701645b8bc497aaf63e970157bdd80]) successfully deleted
I20240502 01:56:08.698168  5905 catalog_manager.cc:4830] TS 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867): tablet 265a9645e1f141cb88c786b9aef8f90e (table rebalance_test_table_7 [id=c2701645b8bc497aaf63e970157bdd80]) successfully deleted
I20240502 01:56:08.699427 12698 tablet_replica.cc:330] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 07cdb213c77c4ffda1f999b66b725399: stopping tablet replica
I20240502 01:56:08.701086 12698 raft_consensus.cc:2229] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:08.701613 12698 raft_consensus.cc:2258] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:08.702664 12698 ts_tablet_manager.cc:1875] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 07cdb213c77c4ffda1f999b66b725399: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:56:08.745015  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:08.772629 12698 ts_tablet_manager.cc:1888] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 07cdb213c77c4ffda1f999b66b725399: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:56:08.779428 12698 log.cc:1196] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 07cdb213c77c4ffda1f999b66b725399: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal/wals/afa5561f8c0e4c0fa9bbbd5da61348c7
I20240502 01:56:08.783092  5905 catalog_manager.cc:4830] TS 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867): tablet afa5561f8c0e4c0fa9bbbd5da61348c7 (table rebalance_test_table_2 [id=ef7ebedfbbff4054b074bf7a7622b632]) successfully deleted
I20240502 01:56:08.795032  6048 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:08.796599  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:08.837735  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:08.838196  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:08.850955  6048 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:08.857712  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:08.880921  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:56:08.902987  8385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:08.926586  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:08.949928  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:08.955854  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:09.008313  8385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:09.028700  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:09.040016  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:56:09.112118  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:09.486389  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:09.510661  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:09.511374  6048 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:09.550145  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:09.575525  8385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:09.587962  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:09.638232  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:09.653349  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:09.701071  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:09.708886  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:56:09.717557  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:09.721323  6048 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:09.739403  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:09.739652  8385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:09.776227  6200 consensus_queue.cc:237] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:56:09.789245  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 01:56:09.790669  6000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.131:44245, user_credentials={real_user=slave}} blocked reactor thread for 59483.4us
I20240502 01:56:09.813275  8405 raft_consensus.cc:1261] T 086509f1ee0a4a81b2f8194b50dbe159 P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Refusing update from remote peer a2c90ee440b44b708bccc2e2d5385cc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:56:09.814230 12560 consensus_queue.cc:1035] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:56:09.815399  6336 consensus_queue.cc:237] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:56:09.823920  8269 raft_consensus.cc:1261] T 086509f1ee0a4a81b2f8194b50dbe159 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Refusing update from remote peer a2c90ee440b44b708bccc2e2d5385cc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:56:09.824716 12560 consensus_queue.cc:1035] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:56:09.828603  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:09.833778  6204 raft_consensus.cc:1261] T 9a5f486852554e2fa2d07f7b9d686ced P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:56:09.834203  8405 raft_consensus.cc:1261] T 9a5f486852554e2fa2d07f7b9d686ced P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:56:09.834463 12424 consensus_queue.cc:1035] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:56:09.834719  6068 raft_consensus.cc:1261] T 086509f1ee0a4a81b2f8194b50dbe159 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Refusing update from remote peer a2c90ee440b44b708bccc2e2d5385cc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:56:09.834800 12687 consensus_queue.cc:1035] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:56:09.835919 12560 consensus_queue.cc:1035] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.013s
I20240502 01:56:09.869067 12424 raft_consensus.cc:2932] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:09.876907  6200 raft_consensus.cc:2932] T 9a5f486852554e2fa2d07f7b9d686ced P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:09.884912  5912 catalog_manager.cc:5484] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 reported cstate change: config changed from index 4 to 5, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:09.897814 12889 raft_consensus.cc:2932] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
W20240502 01:56:09.899382  6134 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.135:40543, user_credentials={real_user=slave}} blocked reactor thread for 56071.4us
I20240502 01:56:09.902963  6068 raft_consensus.cc:2932] T 086509f1ee0a4a81b2f8194b50dbe159 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:09.914220  5920 catalog_manager.cc:5484] T 086509f1ee0a4a81b2f8194b50dbe159 P 218bc8c1f9ce46059898fc738179adf9 reported cstate change: config changed from index 5 to 6, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New cstate: current_term: 1 leader_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:09.935559  8405 raft_consensus.cc:2932] T 9a5f486852554e2fa2d07f7b9d686ced P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:10.013438  8405 raft_consensus.cc:2932] T 086509f1ee0a4a81b2f8194b50dbe159 P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:10.019244  8269 raft_consensus.cc:2932] T 086509f1ee0a4a81b2f8194b50dbe159 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:10.056651  6048 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a5f486852554e2fa2d07f7b9d686ced with delete_type TABLET_DATA_TOMBSTONED (TS 218bc8c1f9ce46059898fc738179adf9 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:50408
I20240502 01:56:10.103261 12908 tablet_replica.cc:330] T 9a5f486852554e2fa2d07f7b9d686ced P 218bc8c1f9ce46059898fc738179adf9: stopping tablet replica
I20240502 01:56:10.103650 12908 raft_consensus.cc:2229] T 9a5f486852554e2fa2d07f7b9d686ced P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:10.107714 12908 raft_consensus.cc:2258] T 9a5f486852554e2fa2d07f7b9d686ced P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:10.113121  6447 tablet_service.cc:1514] Processing DeleteTablet for tablet 086509f1ee0a4a81b2f8194b50dbe159 with delete_type TABLET_DATA_TOMBSTONED (TS 07cdb213c77c4ffda1f999b66b725399 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56564
I20240502 01:56:10.141798 12908 ts_tablet_manager.cc:1875] T 9a5f486852554e2fa2d07f7b9d686ced P 218bc8c1f9ce46059898fc738179adf9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:56:10.174188 12913 tablet_replica.cc:330] T 086509f1ee0a4a81b2f8194b50dbe159 P 07cdb213c77c4ffda1f999b66b725399: stopping tablet replica
I20240502 01:56:10.174448 12913 raft_consensus.cc:2229] T 086509f1ee0a4a81b2f8194b50dbe159 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:10.174772 12913 raft_consensus.cc:2258] T 086509f1ee0a4a81b2f8194b50dbe159 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:10.175611 12913 ts_tablet_manager.cc:1875] T 086509f1ee0a4a81b2f8194b50dbe159 P 07cdb213c77c4ffda1f999b66b725399: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:56:10.206719 12908 ts_tablet_manager.cc:1888] T 9a5f486852554e2fa2d07f7b9d686ced P 218bc8c1f9ce46059898fc738179adf9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:56:10.207415 12908 log.cc:1196] T 9a5f486852554e2fa2d07f7b9d686ced P 218bc8c1f9ce46059898fc738179adf9: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal/wals/9a5f486852554e2fa2d07f7b9d686ced
I20240502 01:56:10.208534  5907 catalog_manager.cc:4830] TS 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227): tablet 9a5f486852554e2fa2d07f7b9d686ced (table rebalance_test_table_0 [id=ef6f7a2e933049e5982deec12e9a670a]) successfully deleted
I20240502 01:56:10.222098 12913 ts_tablet_manager.cc:1888] T 086509f1ee0a4a81b2f8194b50dbe159 P 07cdb213c77c4ffda1f999b66b725399: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:56:10.223380 12913 log.cc:1196] T 086509f1ee0a4a81b2f8194b50dbe159 P 07cdb213c77c4ffda1f999b66b725399: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal/wals/086509f1ee0a4a81b2f8194b50dbe159
I20240502 01:56:10.228399  6048 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:10.250043  5905 catalog_manager.cc:4830] TS 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867): tablet 086509f1ee0a4a81b2f8194b50dbe159 (table rebalance_test_table_0 [id=ef6f7a2e933049e5982deec12e9a670a]) successfully deleted
I20240502 01:56:10.267428  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:10.284005  8385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:10.294389  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:10.311203  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:10.336691  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:56:10.338538  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:10.384819  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:10.402441  6068 consensus_queue.cc:237] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:56:10.414183  8269 raft_consensus.cc:1261] T 99e74a49b42745888058db67049dc682 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:56:10.414995 12409 consensus_queue.cc:1035] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:56:10.417456  6200 raft_consensus.cc:1261] T 99e74a49b42745888058db67049dc682 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:56:10.418359 12409 consensus_queue.cc:1035] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:56:10.452339 12409 raft_consensus.cc:2932] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:10.486826  5912 catalog_manager.cc:5484] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 reported cstate change: config changed from index 4 to 5, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New cstate: current_term: 1 leader_uuid: "218bc8c1f9ce46059898fc738179adf9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:10.498417  6447 tablet_service.cc:1514] Processing DeleteTablet for tablet 99e74a49b42745888058db67049dc682 with delete_type TABLET_DATA_TOMBSTONED (TS 07cdb213c77c4ffda1f999b66b725399 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:56564
I20240502 01:56:10.498838 12913 tablet_replica.cc:330] T 99e74a49b42745888058db67049dc682 P 07cdb213c77c4ffda1f999b66b725399: stopping tablet replica
I20240502 01:56:10.499050 12913 raft_consensus.cc:2229] T 99e74a49b42745888058db67049dc682 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:10.499326 12913 raft_consensus.cc:2258] T 99e74a49b42745888058db67049dc682 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:10.500125 12913 ts_tablet_manager.cc:1875] T 99e74a49b42745888058db67049dc682 P 07cdb213c77c4ffda1f999b66b725399: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:56:10.502084  8270 raft_consensus.cc:2932] T 99e74a49b42745888058db67049dc682 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:10.509238  6200 raft_consensus.cc:2932] T 99e74a49b42745888058db67049dc682 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:10.549480 12913 ts_tablet_manager.cc:1888] T 99e74a49b42745888058db67049dc682 P 07cdb213c77c4ffda1f999b66b725399: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:56:10.552757 12913 log.cc:1196] T 99e74a49b42745888058db67049dc682 P 07cdb213c77c4ffda1f999b66b725399: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal/wals/99e74a49b42745888058db67049dc682
I20240502 01:56:10.554272  5905 catalog_manager.cc:4830] TS 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867): tablet 99e74a49b42745888058db67049dc682 (table rebalance_test_table_8 [id=fcde569cfdf747c2aaecac7d265b4b1f]) successfully deleted
I20240502 01:56:10.608779  6067 consensus_queue.cc:237] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:56:10.626452  8269 raft_consensus.cc:1261] T 5be45a68cff24a75a03b2fcd4b7851f9 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:56:10.627372 12409 consensus_queue.cc:1035] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:56:10.635205  8135 raft_consensus.cc:1261] T 5be45a68cff24a75a03b2fcd4b7851f9 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:56:10.636077 12409 consensus_queue.cc:1035] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:56:10.652201  6336 raft_consensus.cc:1261] T 5be45a68cff24a75a03b2fcd4b7851f9 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:56:10.653210 12409 consensus_queue.cc:1035] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:56:10.666950 12409 raft_consensus.cc:2932] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:10.669111  8269 raft_consensus.cc:2932] T 5be45a68cff24a75a03b2fcd4b7851f9 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:10.673511  8135 raft_consensus.cc:2932] T 5be45a68cff24a75a03b2fcd4b7851f9 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:10.678262  5912 catalog_manager.cc:5484] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 reported cstate change: config changed from index 5 to 6, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New cstate: current_term: 1 leader_uuid: "218bc8c1f9ce46059898fc738179adf9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:10.697664  6180 tablet_service.cc:1514] Processing DeleteTablet for tablet 5be45a68cff24a75a03b2fcd4b7851f9 with delete_type TABLET_DATA_TOMBSTONED (TS a2c90ee440b44b708bccc2e2d5385cc0 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41258
I20240502 01:56:10.701109 13022 tablet_replica.cc:330] T 5be45a68cff24a75a03b2fcd4b7851f9 P a2c90ee440b44b708bccc2e2d5385cc0: stopping tablet replica
I20240502 01:56:10.702088 13022 raft_consensus.cc:2229] T 5be45a68cff24a75a03b2fcd4b7851f9 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:10.703070 13022 raft_consensus.cc:2258] T 5be45a68cff24a75a03b2fcd4b7851f9 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:10.705858 13022 ts_tablet_manager.cc:1875] T 5be45a68cff24a75a03b2fcd4b7851f9 P a2c90ee440b44b708bccc2e2d5385cc0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:56:10.709398  6334 raft_consensus.cc:2932] T 5be45a68cff24a75a03b2fcd4b7851f9 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:10.730818 13022 ts_tablet_manager.cc:1888] T 5be45a68cff24a75a03b2fcd4b7851f9 P a2c90ee440b44b708bccc2e2d5385cc0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:56:10.731041 13022 log.cc:1196] T 5be45a68cff24a75a03b2fcd4b7851f9 P a2c90ee440b44b708bccc2e2d5385cc0: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal/wals/5be45a68cff24a75a03b2fcd4b7851f9
I20240502 01:56:10.732064  5908 catalog_manager.cc:4830] TS a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197): tablet 5be45a68cff24a75a03b2fcd4b7851f9 (table rebalance_test_table_9 [id=f406d4a60a5e40c485796d37ff4d76bc]) successfully deleted
I20240502 01:56:10.891355  6067 consensus_queue.cc:237] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:56:10.893908  8269 raft_consensus.cc:1261] T 99e74a49b42745888058db67049dc682 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:56:10.895080 12877 consensus_queue.cc:1035] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:56:10.898046 12409 raft_consensus.cc:2932] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:10.900732  8270 raft_consensus.cc:2932] T 99e74a49b42745888058db67049dc682 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:10.904676  5921 catalog_manager.cc:5484] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 reported cstate change: config changed from index 5 to 6, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New cstate: current_term: 1 leader_uuid: "218bc8c1f9ce46059898fc738179adf9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:10.913338  6180 tablet_service.cc:1514] Processing DeleteTablet for tablet 99e74a49b42745888058db67049dc682 with delete_type TABLET_DATA_TOMBSTONED (TS a2c90ee440b44b708bccc2e2d5385cc0 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41258
I20240502 01:56:10.913712 13022 tablet_replica.cc:330] T 99e74a49b42745888058db67049dc682 P a2c90ee440b44b708bccc2e2d5385cc0: stopping tablet replica
I20240502 01:56:10.913947 13022 raft_consensus.cc:2229] T 99e74a49b42745888058db67049dc682 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:10.914188 13022 raft_consensus.cc:2258] T 99e74a49b42745888058db67049dc682 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:10.915455 13022 ts_tablet_manager.cc:1875] T 99e74a49b42745888058db67049dc682 P a2c90ee440b44b708bccc2e2d5385cc0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:56:10.918081  6067 consensus_queue.cc:237] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:56:10.921415  8270 raft_consensus.cc:1261] T 99e74a49b42745888058db67049dc682 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 01:56:10.921617  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99e74a49b42745888058db67049dc682. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:56:10.922263 12877 consensus_queue.cc:1035] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:56:10.929574 12409 raft_consensus.cc:2932] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:10.932360  8270 raft_consensus.cc:2932] T 99e74a49b42745888058db67049dc682 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:10.934753  5907 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 99e74a49b42745888058db67049dc682 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 01:56:10.937541  5921 catalog_manager.cc:5484] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 reported cstate change: config changed from index 6 to 7, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New cstate: current_term: 1 leader_uuid: "218bc8c1f9ce46059898fc738179adf9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:10.939793 13022 ts_tablet_manager.cc:1888] T 99e74a49b42745888058db67049dc682 P a2c90ee440b44b708bccc2e2d5385cc0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:56:10.939965 13022 log.cc:1196] T 99e74a49b42745888058db67049dc682 P a2c90ee440b44b708bccc2e2d5385cc0: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal/wals/99e74a49b42745888058db67049dc682
I20240502 01:56:10.940740  5908 catalog_manager.cc:4830] TS a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197): tablet 99e74a49b42745888058db67049dc682 (table rebalance_test_table_8 [id=fcde569cfdf747c2aaecac7d265b4b1f]) successfully deleted
I20240502 01:56:11.018291  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:11.120165  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:11.146407  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:11.151173  6048 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:11.204694  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:11.212296  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:11.239641  6048 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:11.258252  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:11.300937  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:11.349452  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:11.374344  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:11.389835  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:11.413710  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:56:11.440557  8385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:11.449736  8385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:11.466845 13135 ts_tablet_manager.cc:902] T 99e74a49b42745888058db67049dc682 P 9db17f84d37944098cbeb9e0d8456efb: Initiating tablet copy from peer 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227)
I20240502 01:56:11.505686  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:56:11.590817 13135 tablet_copy_client.cc:308] T 99e74a49b42745888058db67049dc682 P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.129:36227
I20240502 01:56:11.601478  6089 tablet_copy_service.cc:140] P 218bc8c1f9ce46059898fc738179adf9: Received BeginTabletCopySession request for tablet 99e74a49b42745888058db67049dc682 from peer 9db17f84d37944098cbeb9e0d8456efb ({username='slave'} at 127.25.109.134:39709)
I20240502 01:56:11.601667  6089 tablet_copy_service.cc:161] P 218bc8c1f9ce46059898fc738179adf9: Beginning new tablet copy session on tablet 99e74a49b42745888058db67049dc682 from peer 9db17f84d37944098cbeb9e0d8456efb at {username='slave'} at 127.25.109.134:39709: session id = 9db17f84d37944098cbeb9e0d8456efb-99e74a49b42745888058db67049dc682
I20240502 01:56:11.605100  6089 tablet_copy_source_session.cc:205] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:56:11.607321 13135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99e74a49b42745888058db67049dc682. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:56:11.679589 13135 tablet_copy_client.cc:783] T 99e74a49b42745888058db67049dc682 P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Starting download of 0 data blocks...
I20240502 01:56:11.680284 13135 tablet_copy_client.cc:647] T 99e74a49b42745888058db67049dc682 P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Starting download of 1 WAL segments...
I20240502 01:56:11.683590 13135 tablet_copy_client.cc:523] T 99e74a49b42745888058db67049dc682 P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:56:11.687440 13135 tablet_bootstrap.cc:492] T 99e74a49b42745888058db67049dc682 P 9db17f84d37944098cbeb9e0d8456efb: Bootstrap starting.
I20240502 01:56:11.784765  6048 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:11.823993  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:11.826280  8385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:11.838410  6465 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "73ed91df7ff04846991c95b30a86ea7e"
dest_uuid: "07cdb213c77c4ffda1f999b66b725399"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:45080
I20240502 01:56:11.838667  6465 raft_consensus.cc:595] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:56:11.856588  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:11.861469  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:11.867233  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:56:11.902631  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:11.967693  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:11.995465  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:12.002663  6048 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:12.011385  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:12.095849  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:12.172395  8385 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:12.184741  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:56:12.193851 13135 tablet_bootstrap.cc:492] T 99e74a49b42745888058db67049dc682 P 9db17f84d37944098cbeb9e0d8456efb: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:56:12.194412 13135 tablet_bootstrap.cc:492] T 99e74a49b42745888058db67049dc682 P 9db17f84d37944098cbeb9e0d8456efb: Bootstrap complete.
I20240502 01:56:12.194727 13135 ts_tablet_manager.cc:1367] T 99e74a49b42745888058db67049dc682 P 9db17f84d37944098cbeb9e0d8456efb: Time spent bootstrapping tablet: real 0.507s	user 0.054s	sys 0.005s
I20240502 01:56:12.200940 13001 raft_consensus.cc:979] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399: : Instructing follower 6d34e50fb5dd4a1db58013d4863b8992 to start an election
I20240502 01:56:12.201079 13001 raft_consensus.cc:1067] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Signalling peer 6d34e50fb5dd4a1db58013d4863b8992 to start an election
I20240502 01:56:12.201985  8405 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "73ed91df7ff04846991c95b30a86ea7e"
dest_uuid: "6d34e50fb5dd4a1db58013d4863b8992"
 from {username='slave'} at 127.25.109.132:41331
I20240502 01:56:12.202363  8405 raft_consensus.cc:482] T 73ed91df7ff04846991c95b30a86ea7e P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:56:12.202584  8405 raft_consensus.cc:3037] T 73ed91df7ff04846991c95b30a86ea7e P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:56:12.207369  8405 raft_consensus.cc:504] T 73ed91df7ff04846991c95b30a86ea7e P 6d34e50fb5dd4a1db58013d4863b8992 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:56:12.210209 13135 raft_consensus.cc:348] T 99e74a49b42745888058db67049dc682 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:56:12.212023  6067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ed91df7ff04846991c95b30a86ea7e" candidate_uuid: "6d34e50fb5dd4a1db58013d4863b8992" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "218bc8c1f9ce46059898fc738179adf9"
I20240502 01:56:12.212337  6067 raft_consensus.cc:3037] T 73ed91df7ff04846991c95b30a86ea7e P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:56:12.210502 13135 raft_consensus.cc:729] T 99e74a49b42745888058db67049dc682 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9db17f84d37944098cbeb9e0d8456efb, State: Initialized, Role: FOLLOWER
I20240502 01:56:12.219281 13135 consensus_queue.cc:260] T 99e74a49b42745888058db67049dc682 P 9db17f84d37944098cbeb9e0d8456efb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:56:12.220181  6067 raft_consensus.cc:2454] T 73ed91df7ff04846991c95b30a86ea7e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d34e50fb5dd4a1db58013d4863b8992 in term 2.
I20240502 01:56:12.229892  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:12.233197  6200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ed91df7ff04846991c95b30a86ea7e" candidate_uuid: "6d34e50fb5dd4a1db58013d4863b8992" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "a2c90ee440b44b708bccc2e2d5385cc0"
I20240502 01:56:12.233484  6200 raft_consensus.cc:3037] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:56:12.237783  6200 raft_consensus.cc:2454] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d34e50fb5dd4a1db58013d4863b8992 in term 2.
I20240502 01:56:12.238423  8338 leader_election.cc:302] T 73ed91df7ff04846991c95b30a86ea7e P 6d34e50fb5dd4a1db58013d4863b8992 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9, 6d34e50fb5dd4a1db58013d4863b8992, a2c90ee440b44b708bccc2e2d5385cc0; no voters: 
I20240502 01:56:12.238792 12660 raft_consensus.cc:2781] T 73ed91df7ff04846991c95b30a86ea7e P 6d34e50fb5dd4a1db58013d4863b8992 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:56:12.240761 13135 ts_tablet_manager.cc:1398] T 99e74a49b42745888058db67049dc682 P 9db17f84d37944098cbeb9e0d8456efb: Time spent starting tablet: real 0.046s	user 0.004s	sys 0.001s
I20240502 01:56:12.242393  6089 tablet_copy_service.cc:342] P 218bc8c1f9ce46059898fc738179adf9: Request end of tablet copy session 9db17f84d37944098cbeb9e0d8456efb-99e74a49b42745888058db67049dc682 received from {username='slave'} at 127.25.109.134:39709
I20240502 01:56:12.242518  6089 tablet_copy_service.cc:433] P 218bc8c1f9ce46059898fc738179adf9: ending tablet copy session 9db17f84d37944098cbeb9e0d8456efb-99e74a49b42745888058db67049dc682 on tablet 99e74a49b42745888058db67049dc682 with peer 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:56:12.251369  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:12.251744  8405 leader_election.cc:288] T 73ed91df7ff04846991c95b30a86ea7e P 6d34e50fb5dd4a1db58013d4863b8992 [CANDIDATE]: Term 2 election: Requested vote from peers 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227), a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197), 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867), 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135:40543)
I20240502 01:56:12.253002  6465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ed91df7ff04846991c95b30a86ea7e" candidate_uuid: "6d34e50fb5dd4a1db58013d4863b8992" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "07cdb213c77c4ffda1f999b66b725399"
I20240502 01:56:12.253222  6465 raft_consensus.cc:3032] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:56:12.253283  6465 raft_consensus.cc:729] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 07cdb213c77c4ffda1f999b66b725399, State: Running, Role: LEADER
I20240502 01:56:12.253563  6465 consensus_queue.cc:260] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:56:12.254029  6465 raft_consensus.cc:3037] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:56:12.258944  6465 raft_consensus.cc:2454] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d34e50fb5dd4a1db58013d4863b8992 in term 2.
I20240502 01:56:12.271755 12660 raft_consensus.cc:686] T 73ed91df7ff04846991c95b30a86ea7e P 6d34e50fb5dd4a1db58013d4863b8992 [term 2 LEADER]: Becoming Leader. State: Replica: 6d34e50fb5dd4a1db58013d4863b8992, State: Running, Role: LEADER
I20240502 01:56:12.272361 12660 consensus_queue.cc:237] T 73ed91df7ff04846991c95b30a86ea7e P 6d34e50fb5dd4a1db58013d4863b8992 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:56:12.304313  6465 consensus_queue.cc:237] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 01:56:12.306667  5916 catalog_manager.cc:5484] T 73ed91df7ff04846991c95b30a86ea7e P 6d34e50fb5dd4a1db58013d4863b8992 reported cstate change: term changed from 1 to 2, leader changed from 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) to 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136). New cstate: current_term: 2 leader_uuid: "6d34e50fb5dd4a1db58013d4863b8992" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:12.311853  8270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ed91df7ff04846991c95b30a86ea7e" candidate_uuid: "6d34e50fb5dd4a1db58013d4863b8992" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "436822a397664c54b8d1b9eb3bc6cff4"
I20240502 01:56:12.312278  8270 raft_consensus.cc:3037] T 73ed91df7ff04846991c95b30a86ea7e P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:56:12.315495  8270 raft_consensus.cc:2454] T 73ed91df7ff04846991c95b30a86ea7e P 436822a397664c54b8d1b9eb3bc6cff4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d34e50fb5dd4a1db58013d4863b8992 in term 2.
I20240502 01:56:12.324007  7998 raft_consensus.cc:1261] T 0cecd1b14e474dee825fae040ce1b31e P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:56:12.325248 13136 consensus_queue.cc:1035] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:56:12.333499  8135 raft_consensus.cc:1261] T 0cecd1b14e474dee825fae040ce1b31e P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:56:12.334579  6336 raft_consensus.cc:1261] T 0cecd1b14e474dee825fae040ce1b31e P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:56:12.336556 13001 consensus_queue.cc:1035] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:56:12.337275 13001 consensus_queue.cc:1035] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:56:12.349503  8135 raft_consensus.cc:1203] T 99e74a49b42745888058db67049dc682 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 01:56:12.378298 13255 raft_consensus.cc:2932] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:56:12.380388  8135 raft_consensus.cc:2932] T 0cecd1b14e474dee825fae040ce1b31e P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:56:12.390668  6336 raft_consensus.cc:2932] T 0cecd1b14e474dee825fae040ce1b31e P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:56:12.391263  5916 catalog_manager.cc:5484] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 reported cstate change: config changed from index 5 to 6, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New cstate: current_term: 1 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:56:12.396541  7998 raft_consensus.cc:2932] T 0cecd1b14e474dee825fae040ce1b31e P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:56:12.412014  6048 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cecd1b14e474dee825fae040ce1b31e with delete_type TABLET_DATA_TOMBSTONED (TS 218bc8c1f9ce46059898fc738179adf9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50408
I20240502 01:56:12.428994 13271 tablet_replica.cc:330] T 0cecd1b14e474dee825fae040ce1b31e P 218bc8c1f9ce46059898fc738179adf9: stopping tablet replica
I20240502 01:56:12.429370 13271 raft_consensus.cc:2229] T 0cecd1b14e474dee825fae040ce1b31e P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:12.429805 13271 raft_consensus.cc:2258] T 0cecd1b14e474dee825fae040ce1b31e P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:12.430712 13271 ts_tablet_manager.cc:1875] T 0cecd1b14e474dee825fae040ce1b31e P 218bc8c1f9ce46059898fc738179adf9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:56:12.503351 13271 ts_tablet_manager.cc:1888] T 0cecd1b14e474dee825fae040ce1b31e P 218bc8c1f9ce46059898fc738179adf9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:56:12.503706 13271 log.cc:1196] T 0cecd1b14e474dee825fae040ce1b31e P 218bc8c1f9ce46059898fc738179adf9: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal/wals/0cecd1b14e474dee825fae040ce1b31e
I20240502 01:56:12.504988  5907 catalog_manager.cc:4830] TS 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227): tablet 0cecd1b14e474dee825fae040ce1b31e (table rebalance_test_table_0 [id=ef6f7a2e933049e5982deec12e9a670a]) successfully deleted
I20240502 01:56:12.651993  6447 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:12.684963  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:12.699980  8270 raft_consensus.cc:1261] T 73ed91df7ff04846991c95b30a86ea7e P 436822a397664c54b8d1b9eb3bc6cff4 [term 2 FOLLOWER]: Refusing update from remote peer 6d34e50fb5dd4a1db58013d4863b8992: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:56:12.700750 12660 consensus_queue.cc:1035] T 73ed91df7ff04846991c95b30a86ea7e P 6d34e50fb5dd4a1db58013d4863b8992 [LEADER]: Connected to new peer: Peer: permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:56:12.736028  6465 raft_consensus.cc:1261] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399 [term 2 FOLLOWER]: Refusing update from remote peer 6d34e50fb5dd4a1db58013d4863b8992: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:56:12.741082  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:12.741616 12660 consensus_queue.cc:1035] T 73ed91df7ff04846991c95b30a86ea7e P 6d34e50fb5dd4a1db58013d4863b8992 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:56:12.758208  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:12.799539  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:12.825963  6048 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:12.833140  8385 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:56:12.857792  6200 raft_consensus.cc:1261] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 [term 2 FOLLOWER]: Refusing update from remote peer 6d34e50fb5dd4a1db58013d4863b8992: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:56:12.858688 12660 consensus_queue.cc:1035] T 73ed91df7ff04846991c95b30a86ea7e P 6d34e50fb5dd4a1db58013d4863b8992 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:56:12.883419  6465 consensus_queue.cc:237] T 0039dd405cfc4dcea5e7353169e3cd52 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 01:56:12.897351  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:56:12.906808  6067 raft_consensus.cc:1261] T 73ed91df7ff04846991c95b30a86ea7e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Refusing update from remote peer 6d34e50fb5dd4a1db58013d4863b8992: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:56:12.920640  8001 raft_consensus.cc:1261] T 0039dd405cfc4dcea5e7353169e3cd52 P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:56:12.921455 13255 consensus_queue.cc:1035] T 0039dd405cfc4dcea5e7353169e3cd52 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:56:12.913172 12660 consensus_queue.cc:1035] T 73ed91df7ff04846991c95b30a86ea7e P 6d34e50fb5dd4a1db58013d4863b8992 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:56:12.951310  6203 raft_consensus.cc:1261] T 0039dd405cfc4dcea5e7353169e3cd52 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:56:12.957592 13255 consensus_queue.cc:1035] T 0039dd405cfc4dcea5e7353169e3cd52 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:56:12.963508 13255 raft_consensus.cc:2932] T 0039dd405cfc4dcea5e7353169e3cd52 P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:56:12.971226  6203 raft_consensus.cc:2932] T 0039dd405cfc4dcea5e7353169e3cd52 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:56:12.978461  5921 catalog_manager.cc:5484] T 0039dd405cfc4dcea5e7353169e3cd52 P 07cdb213c77c4ffda1f999b66b725399 reported cstate change: config changed from index 4 to 5, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New cstate: current_term: 1 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
W20240502 01:56:12.991604  8337 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.129:36227, user_credentials={real_user=slave}} blocked reactor thread for 83278.4us
I20240502 01:56:12.998574  7998 raft_consensus.cc:2932] T 0039dd405cfc4dcea5e7353169e3cd52 P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:56:13.025919  6447 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:13.043941  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:13.069438  6316 tablet_service.cc:1514] Processing DeleteTablet for tablet 0039dd405cfc4dcea5e7353169e3cd52 with delete_type TABLET_DATA_TOMBSTONED (TS 94ffd7aaf85b4e17881241b8396fa2f3 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41134
I20240502 01:56:13.098273  6048 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:13.105605  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:13.112144  8385 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:56:13.129645  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:13.132665 13400 tablet_replica.cc:330] T 0039dd405cfc4dcea5e7353169e3cd52 P 94ffd7aaf85b4e17881241b8396fa2f3: stopping tablet replica
I20240502 01:56:13.135234 13400 raft_consensus.cc:2229] T 0039dd405cfc4dcea5e7353169e3cd52 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:13.135604 13400 raft_consensus.cc:2258] T 0039dd405cfc4dcea5e7353169e3cd52 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:13.137902 13400 ts_tablet_manager.cc:1875] T 0039dd405cfc4dcea5e7353169e3cd52 P 94ffd7aaf85b4e17881241b8396fa2f3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:56:13.147223  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:56:13.202713 13400 ts_tablet_manager.cc:1888] T 0039dd405cfc4dcea5e7353169e3cd52 P 94ffd7aaf85b4e17881241b8396fa2f3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:56:13.202947 13400 log.cc:1196] T 0039dd405cfc4dcea5e7353169e3cd52 P 94ffd7aaf85b4e17881241b8396fa2f3: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/wals/0039dd405cfc4dcea5e7353169e3cd52
I20240502 01:56:13.205381  5907 catalog_manager.cc:4830] TS 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245): tablet 0039dd405cfc4dcea5e7353169e3cd52 (table rebalance_test_table_7 [id=c2701645b8bc497aaf63e970157bdd80]) successfully deleted
I20240502 01:56:13.213975  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:13.526202  6066 consensus_queue.cc:237] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:56:13.539587  8405 raft_consensus.cc:1261] T c58d6b2401c047cd9af56a66c2e00105 P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:56:13.540452 13134 consensus_queue.cc:1035] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:56:13.540664  8135 raft_consensus.cc:1261] T c58d6b2401c047cd9af56a66c2e00105 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:56:13.543309 13134 consensus_queue.cc:1035] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:56:13.545586 13134 raft_consensus.cc:2932] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:13.547781  8405 raft_consensus.cc:2932] T c58d6b2401c047cd9af56a66c2e00105 P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:13.600471  5921 catalog_manager.cc:5484] T c58d6b2401c047cd9af56a66c2e00105 P 6d34e50fb5dd4a1db58013d4863b8992 reported cstate change: config changed from index 6 to 7, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New cstate: current_term: 1 leader_uuid: "218bc8c1f9ce46059898fc738179adf9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:13.633203  6447 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 01:56:13.633482  5999 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.136:44359, user_credentials={real_user=slave}} blocked reactor thread for 70067.5us
W20240502 01:56:13.637133  6000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.134:42517, user_credentials={real_user=slave}} blocked reactor thread for 70107.9us
I20240502 01:56:13.638511  8135 raft_consensus.cc:2932] T c58d6b2401c047cd9af56a66c2e00105 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:13.680673  6447 tablet_service.cc:1514] Processing DeleteTablet for tablet c58d6b2401c047cd9af56a66c2e00105 with delete_type TABLET_DATA_TOMBSTONED (TS 07cdb213c77c4ffda1f999b66b725399 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:56564
I20240502 01:56:13.701387 13524 tablet_replica.cc:330] T c58d6b2401c047cd9af56a66c2e00105 P 07cdb213c77c4ffda1f999b66b725399: stopping tablet replica
I20240502 01:56:13.713757 13524 raft_consensus.cc:2229] T c58d6b2401c047cd9af56a66c2e00105 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:13.717787 13524 raft_consensus.cc:2258] T c58d6b2401c047cd9af56a66c2e00105 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:13.719478  8385 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:56:13.723001 13524 ts_tablet_manager.cc:1875] T c58d6b2401c047cd9af56a66c2e00105 P 07cdb213c77c4ffda1f999b66b725399: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:56:13.724931  6048 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:13.745589  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:13.763578 13524 ts_tablet_manager.cc:1888] T c58d6b2401c047cd9af56a66c2e00105 P 07cdb213c77c4ffda1f999b66b725399: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:56:13.776202  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:56:13.798606 13524 log.cc:1196] T c58d6b2401c047cd9af56a66c2e00105 P 07cdb213c77c4ffda1f999b66b725399: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal/wals/c58d6b2401c047cd9af56a66c2e00105
I20240502 01:56:13.821241  5905 catalog_manager.cc:4830] TS 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867): tablet c58d6b2401c047cd9af56a66c2e00105 (table rebalance_test_table_4 [id=1ed4a88787d6401481245c552170a456]) successfully deleted
I20240502 01:56:13.871712  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:13.902706  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:13.918005  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:13.988188  8405 consensus_queue.cc:237] T 73ed91df7ff04846991c95b30a86ea7e P 6d34e50fb5dd4a1db58013d4863b8992 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:56:13.991889  6200 raft_consensus.cc:1261] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 [term 2 FOLLOWER]: Refusing update from remote peer 6d34e50fb5dd4a1db58013d4863b8992: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:56:13.992038  8270 raft_consensus.cc:1261] T 73ed91df7ff04846991c95b30a86ea7e P 436822a397664c54b8d1b9eb3bc6cff4 [term 2 FOLLOWER]: Refusing update from remote peer 6d34e50fb5dd4a1db58013d4863b8992: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:56:13.992687 13483 consensus_queue.cc:1035] T 73ed91df7ff04846991c95b30a86ea7e P 6d34e50fb5dd4a1db58013d4863b8992 [LEADER]: Connected to new peer: Peer: permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:56:13.992853  6066 raft_consensus.cc:1261] T 73ed91df7ff04846991c95b30a86ea7e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Refusing update from remote peer 6d34e50fb5dd4a1db58013d4863b8992: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:56:13.993930 13365 consensus_queue.cc:1035] T 73ed91df7ff04846991c95b30a86ea7e P 6d34e50fb5dd4a1db58013d4863b8992 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:56:13.994544 13365 consensus_queue.cc:1035] T 73ed91df7ff04846991c95b30a86ea7e P 6d34e50fb5dd4a1db58013d4863b8992 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:56:14.003126 13483 raft_consensus.cc:2932] T 73ed91df7ff04846991c95b30a86ea7e P 6d34e50fb5dd4a1db58013d4863b8992 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:14.008827  6066 raft_consensus.cc:2932] T 73ed91df7ff04846991c95b30a86ea7e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:14.020759  5921 catalog_manager.cc:5484] T 73ed91df7ff04846991c95b30a86ea7e P 218bc8c1f9ce46059898fc738179adf9 reported cstate change: config changed from index 5 to 7, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New cstate: current_term: 2 leader_uuid: "6d34e50fb5dd4a1db58013d4863b8992" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:14.031657  6447 tablet_service.cc:1514] Processing DeleteTablet for tablet 73ed91df7ff04846991c95b30a86ea7e with delete_type TABLET_DATA_TOMBSTONED (TS 07cdb213c77c4ffda1f999b66b725399 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:56564
I20240502 01:56:14.032100 13524 tablet_replica.cc:330] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399: stopping tablet replica
I20240502 01:56:14.032316 13524 raft_consensus.cc:2229] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:14.032517 13524 raft_consensus.cc:2258] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:14.033169 13524 ts_tablet_manager.cc:1875] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:56:14.044530 13524 ts_tablet_manager.cc:1888] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 01:56:14.044862 13524 log.cc:1196] T 73ed91df7ff04846991c95b30a86ea7e P 07cdb213c77c4ffda1f999b66b725399: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal/wals/73ed91df7ff04846991c95b30a86ea7e
I20240502 01:56:14.045828  5905 catalog_manager.cc:4830] TS 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867): tablet 73ed91df7ff04846991c95b30a86ea7e (table rebalance_test_table_6 [id=5606c3da105b4b86868c6c51ecdd39aa]) successfully deleted
I20240502 01:56:14.059150  6200 raft_consensus.cc:2932] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:14.084398  8270 raft_consensus.cc:2932] T 73ed91df7ff04846991c95b30a86ea7e P 436822a397664c54b8d1b9eb3bc6cff4 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:14.136933  6048 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:14.191010  6048 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:14.201751  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:14.213603  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:14.227139  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:14.242718  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:14.243724  6447 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:14.249593  8385 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:56:14.250830  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:56:14.280436  6447 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:14.294970  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:14.305136  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:14.342157  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:56:14.393380  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:14.401918  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:14.414597  8385 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:56:14.711544  6465 consensus_queue.cc:237] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:56:14.729036  6203 raft_consensus.cc:1261] T 4eb91f5df851459b8ecf09cbe34fefa5 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:56:14.729012  8135 raft_consensus.cc:1261] T 4eb91f5df851459b8ecf09cbe34fefa5 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:56:14.729782  8405 raft_consensus.cc:1261] T 4eb91f5df851459b8ecf09cbe34fefa5 P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:56:14.729820 13645 consensus_queue.cc:1035] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:56:14.730880 13642 consensus_queue.cc:1035] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:56:14.731472 13642 consensus_queue.cc:1035] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:56:14.754758 13645 raft_consensus.cc:2932] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:14.761865  8406 raft_consensus.cc:2932] T 4eb91f5df851459b8ecf09cbe34fefa5 P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:14.767544  8135 raft_consensus.cc:2932] T 4eb91f5df851459b8ecf09cbe34fefa5 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:14.766680  5914 catalog_manager.cc:5484] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 reported cstate change: config changed from index 5 to 6, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New cstate: current_term: 1 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:14.791229  6203 raft_consensus.cc:2932] T 4eb91f5df851459b8ecf09cbe34fefa5 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:14.799331  6048 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:14.814004  6447 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:14.841807  6316 tablet_service.cc:1514] Processing DeleteTablet for tablet 4eb91f5df851459b8ecf09cbe34fefa5 with delete_type TABLET_DATA_TOMBSTONED (TS 94ffd7aaf85b4e17881241b8396fa2f3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41134
I20240502 01:56:14.847718 13715 tablet_replica.cc:330] T 4eb91f5df851459b8ecf09cbe34fefa5 P 94ffd7aaf85b4e17881241b8396fa2f3: stopping tablet replica
I20240502 01:56:14.849650 13715 raft_consensus.cc:2229] T 4eb91f5df851459b8ecf09cbe34fefa5 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:14.850795 13715 raft_consensus.cc:2258] T 4eb91f5df851459b8ecf09cbe34fefa5 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:14.853449  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:14.853504 13715 ts_tablet_manager.cc:1875] T 4eb91f5df851459b8ecf09cbe34fefa5 P 94ffd7aaf85b4e17881241b8396fa2f3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:56:14.860440  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:14.870246 13715 ts_tablet_manager.cc:1888] T 4eb91f5df851459b8ecf09cbe34fefa5 P 94ffd7aaf85b4e17881241b8396fa2f3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:56:14.870424 13715 log.cc:1196] T 4eb91f5df851459b8ecf09cbe34fefa5 P 94ffd7aaf85b4e17881241b8396fa2f3: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/wals/4eb91f5df851459b8ecf09cbe34fefa5
I20240502 01:56:14.883949  5907 catalog_manager.cc:4830] TS 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245): tablet 4eb91f5df851459b8ecf09cbe34fefa5 (table rebalance_test_table_1 [id=0053d72412224a79be83b7e007b51934]) successfully deleted
I20240502 01:56:14.888777  8385 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:56:14.895115  6203 consensus_queue.cc:237] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:56:14.925091  8135 raft_consensus.cc:1261] T ca458907378e42d48adaa0ee647db4cd P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Refusing update from remote peer a2c90ee440b44b708bccc2e2d5385cc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:56:14.925807 12988 consensus_queue.cc:1035] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:56:14.926762  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:56:14.929163  8406 raft_consensus.cc:1261] T ca458907378e42d48adaa0ee647db4cd P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Refusing update from remote peer a2c90ee440b44b708bccc2e2d5385cc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:56:14.930025 12988 consensus_queue.cc:1035] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:56:14.954958 12988 raft_consensus.cc:2932] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:14.960954  8406 raft_consensus.cc:2932] T ca458907378e42d48adaa0ee647db4cd P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:14.965595  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:15.002786  5914 catalog_manager.cc:5484] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 reported cstate change: config changed from index 6 to 7, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New cstate: current_term: 1 leader_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:15.003057  8134 raft_consensus.cc:2932] T ca458907378e42d48adaa0ee647db4cd P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:15.008034  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:15.113371  6048 tablet_service.cc:1514] Processing DeleteTablet for tablet ca458907378e42d48adaa0ee647db4cd with delete_type TABLET_DATA_TOMBSTONED (TS 218bc8c1f9ce46059898fc738179adf9 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:50408
I20240502 01:56:15.121021 13739 tablet_replica.cc:330] T ca458907378e42d48adaa0ee647db4cd P 218bc8c1f9ce46059898fc738179adf9: stopping tablet replica
I20240502 01:56:15.121320 13739 raft_consensus.cc:2229] T ca458907378e42d48adaa0ee647db4cd P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:15.121577 13739 raft_consensus.cc:2258] T ca458907378e42d48adaa0ee647db4cd P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:15.122432 13739 ts_tablet_manager.cc:1875] T ca458907378e42d48adaa0ee647db4cd P 218bc8c1f9ce46059898fc738179adf9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:56:15.216974 13739 ts_tablet_manager.cc:1888] T ca458907378e42d48adaa0ee647db4cd P 218bc8c1f9ce46059898fc738179adf9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:56:15.217290 13739 log.cc:1196] T ca458907378e42d48adaa0ee647db4cd P 218bc8c1f9ce46059898fc738179adf9: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal/wals/ca458907378e42d48adaa0ee647db4cd
I20240502 01:56:15.218349  5907 catalog_manager.cc:4830] TS 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227): tablet ca458907378e42d48adaa0ee647db4cd (table rebalance_test_table_2 [id=ef7ebedfbbff4054b074bf7a7622b632]) successfully deleted
I20240502 01:56:15.267422  6447 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:15.321171  6048 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:15.344888  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:15.427325  8385 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:56:15.450589  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:15.470513  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:56:15.478271  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:15.567188  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:15.684320  8406 consensus_queue.cc:237] T 265a9645e1f141cb88c786b9aef8f90e P 6d34e50fb5dd4a1db58013d4863b8992 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:56:15.704630  8134 raft_consensus.cc:1261] T 265a9645e1f141cb88c786b9aef8f90e P 9db17f84d37944098cbeb9e0d8456efb [term 2 FOLLOWER]: Refusing update from remote peer 6d34e50fb5dd4a1db58013d4863b8992: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 01:56:15.704705  6066 raft_consensus.cc:1261] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Refusing update from remote peer 6d34e50fb5dd4a1db58013d4863b8992: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 01:56:15.706117 13839 consensus_queue.cc:1035] T 265a9645e1f141cb88c786b9aef8f90e P 6d34e50fb5dd4a1db58013d4863b8992 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:56:15.708104 13571 consensus_queue.cc:1035] T 265a9645e1f141cb88c786b9aef8f90e P 6d34e50fb5dd4a1db58013d4863b8992 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:56:15.727563  6447 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:15.768996 13839 raft_consensus.cc:2932] T 265a9645e1f141cb88c786b9aef8f90e P 6d34e50fb5dd4a1db58013d4863b8992 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:15.775038  8134 raft_consensus.cc:2932] T 265a9645e1f141cb88c786b9aef8f90e P 9db17f84d37944098cbeb9e0d8456efb [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:15.777133  5920 catalog_manager.cc:5484] T 265a9645e1f141cb88c786b9aef8f90e P 6d34e50fb5dd4a1db58013d4863b8992 reported cstate change: config changed from index 7 to 8, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New cstate: current_term: 2 leader_uuid: "6d34e50fb5dd4a1db58013d4863b8992" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:15.813081  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:15.819526  6067 raft_consensus.cc:2932] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:15.822316  6048 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:15.850076  6316 tablet_service.cc:1514] Processing DeleteTablet for tablet 265a9645e1f141cb88c786b9aef8f90e with delete_type TABLET_DATA_TOMBSTONED (TS 94ffd7aaf85b4e17881241b8396fa2f3 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:41134
I20240502 01:56:15.851739  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:15.866729 13868 tablet_replica.cc:330] T 265a9645e1f141cb88c786b9aef8f90e P 94ffd7aaf85b4e17881241b8396fa2f3: stopping tablet replica
I20240502 01:56:15.868299 13868 raft_consensus.cc:2229] T 265a9645e1f141cb88c786b9aef8f90e P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:15.868551 13868 raft_consensus.cc:2258] T 265a9645e1f141cb88c786b9aef8f90e P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:15.869292 13868 ts_tablet_manager.cc:1875] T 265a9645e1f141cb88c786b9aef8f90e P 94ffd7aaf85b4e17881241b8396fa2f3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:56:15.880795  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:56:15.899746 13868 ts_tablet_manager.cc:1888] T 265a9645e1f141cb88c786b9aef8f90e P 94ffd7aaf85b4e17881241b8396fa2f3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240502 01:56:15.899920 13868 log.cc:1196] T 265a9645e1f141cb88c786b9aef8f90e P 94ffd7aaf85b4e17881241b8396fa2f3: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/wals/265a9645e1f141cb88c786b9aef8f90e
I20240502 01:56:15.900743  5907 catalog_manager.cc:4830] TS 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245): tablet 265a9645e1f141cb88c786b9aef8f90e (table rebalance_test_table_7 [id=c2701645b8bc497aaf63e970157bdd80]) successfully deleted
I20240502 01:56:15.908620  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:15.956671  8385 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:56:15.974320  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 01:56:15.980129  6000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.134:42517, user_credentials={real_user=slave}} blocked reactor thread for 47812.1us
W20240502 01:56:15.992707  6402 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.129:36227, user_credentials={real_user=slave}} blocked reactor thread for 72503.1us
I20240502 01:56:16.172745  6067 consensus_queue.cc:237] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:56:16.206553  6447 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:16.212662  8270 raft_consensus.cc:1261] T 5be45a68cff24a75a03b2fcd4b7851f9 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:56:16.213335  8134 raft_consensus.cc:1261] T 5be45a68cff24a75a03b2fcd4b7851f9 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:56:16.213944 13878 consensus_queue.cc:1035] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:56:16.215032 13878 consensus_queue.cc:1035] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:56:16.221915  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:16.227110  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:16.235878 13878 raft_consensus.cc:2932] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:16.237691  8269 raft_consensus.cc:2932] T 5be45a68cff24a75a03b2fcd4b7851f9 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:16.254024  6048 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:16.254065  5912 catalog_manager.cc:5484] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 reported cstate change: config changed from index 6 to 7, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New cstate: current_term: 1 leader_uuid: "218bc8c1f9ce46059898fc738179adf9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:16.261392  6316 tablet_service.cc:1514] Processing DeleteTablet for tablet 5be45a68cff24a75a03b2fcd4b7851f9 with delete_type TABLET_DATA_TOMBSTONED (TS 94ffd7aaf85b4e17881241b8396fa2f3 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:41134
I20240502 01:56:16.261736 13868 tablet_replica.cc:330] T 5be45a68cff24a75a03b2fcd4b7851f9 P 94ffd7aaf85b4e17881241b8396fa2f3: stopping tablet replica
I20240502 01:56:16.261914 13868 raft_consensus.cc:2229] T 5be45a68cff24a75a03b2fcd4b7851f9 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:16.262219 13868 raft_consensus.cc:2258] T 5be45a68cff24a75a03b2fcd4b7851f9 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:16.262974 13868 ts_tablet_manager.cc:1875] T 5be45a68cff24a75a03b2fcd4b7851f9 P 94ffd7aaf85b4e17881241b8396fa2f3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:56:16.267197  8134 raft_consensus.cc:2932] T 5be45a68cff24a75a03b2fcd4b7851f9 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:16.269605 13868 ts_tablet_manager.cc:1888] T 5be45a68cff24a75a03b2fcd4b7851f9 P 94ffd7aaf85b4e17881241b8396fa2f3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:56:16.269820 13868 log.cc:1196] T 5be45a68cff24a75a03b2fcd4b7851f9 P 94ffd7aaf85b4e17881241b8396fa2f3: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/wals/5be45a68cff24a75a03b2fcd4b7851f9
I20240502 01:56:16.270726  5907 catalog_manager.cc:4830] TS 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245): tablet 5be45a68cff24a75a03b2fcd4b7851f9 (table rebalance_test_table_9 [id=f406d4a60a5e40c485796d37ff4d76bc]) successfully deleted
I20240502 01:56:16.304255  8385 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:56:16.320633  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:16.374945  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:56:16.380429  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:16.450314  6447 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:16.485401  8249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:16.505270  6048 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:16.524155  8385 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:56:16.557797  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:16.595764  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:56:16.670526  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:16.700490  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:16.794090  6335 consensus_queue.cc:237] T 55de74ae85e541a19008fd2fc2da3ffd P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:56:16.800143  8269 raft_consensus.cc:1261] T 55de74ae85e541a19008fd2fc2da3ffd P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:56:16.801172  6465 raft_consensus.cc:1261] T 55de74ae85e541a19008fd2fc2da3ffd P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:56:16.802021 13224 consensus_queue.cc:1035] T 55de74ae85e541a19008fd2fc2da3ffd P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:56:16.803391 13224 consensus_queue.cc:1035] T 55de74ae85e541a19008fd2fc2da3ffd P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:56:16.820153  6335 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "89f7cf12d85442749e7d5b259ebca43f"
dest_uuid: "94ffd7aaf85b4e17881241b8396fa2f3"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:38050
I20240502 01:56:16.820505  6335 raft_consensus.cc:595] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:56:16.866814 13224 raft_consensus.cc:2932] T 55de74ae85e541a19008fd2fc2da3ffd P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:16.872500  6465 raft_consensus.cc:2932] T 55de74ae85e541a19008fd2fc2da3ffd P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:16.873652  8269 raft_consensus.cc:2932] T 55de74ae85e541a19008fd2fc2da3ffd P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:16.877930  5914 catalog_manager.cc:5484] T 55de74ae85e541a19008fd2fc2da3ffd P 94ffd7aaf85b4e17881241b8396fa2f3 reported cstate change: config changed from index 4 to 5, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:16.933247  6048 tablet_service.cc:1514] Processing DeleteTablet for tablet 55de74ae85e541a19008fd2fc2da3ffd with delete_type TABLET_DATA_TOMBSTONED (TS 218bc8c1f9ce46059898fc738179adf9 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:50408
I20240502 01:56:16.945596 14026 tablet_replica.cc:330] T 55de74ae85e541a19008fd2fc2da3ffd P 218bc8c1f9ce46059898fc738179adf9: stopping tablet replica
I20240502 01:56:16.945952 14026 raft_consensus.cc:2229] T 55de74ae85e541a19008fd2fc2da3ffd P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:16.946255 14026 raft_consensus.cc:2258] T 55de74ae85e541a19008fd2fc2da3ffd P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:16.947209 14026 ts_tablet_manager.cc:1875] T 55de74ae85e541a19008fd2fc2da3ffd P 218bc8c1f9ce46059898fc738179adf9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:56:16.957444 14026 ts_tablet_manager.cc:1888] T 55de74ae85e541a19008fd2fc2da3ffd P 218bc8c1f9ce46059898fc738179adf9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:56:16.957707 14026 log.cc:1196] T 55de74ae85e541a19008fd2fc2da3ffd P 218bc8c1f9ce46059898fc738179adf9: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal/wals/55de74ae85e541a19008fd2fc2da3ffd
I20240502 01:56:16.958705  5907 catalog_manager.cc:4830] TS 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227): tablet 55de74ae85e541a19008fd2fc2da3ffd (table rebalance_test_table_4 [id=1ed4a88787d6401481245c552170a456]) successfully deleted
I20240502 01:56:17.101045  6048 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:17.105010  8385 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:56:17.138738  6447 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:17.167342  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:17.171020  6316 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:56:17.179008 13224 raft_consensus.cc:979] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3: : Instructing follower 436822a397664c54b8d1b9eb3bc6cff4 to start an election
I20240502 01:56:17.179203 13224 raft_consensus.cc:1067] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Signalling peer 436822a397664c54b8d1b9eb3bc6cff4 to start an election
I20240502 01:56:17.181016  8269 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "89f7cf12d85442749e7d5b259ebca43f"
dest_uuid: "436822a397664c54b8d1b9eb3bc6cff4"
 from {username='slave'} at 127.25.109.131:58337
I20240502 01:56:17.181385  8269 raft_consensus.cc:482] T 89f7cf12d85442749e7d5b259ebca43f P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:56:17.181571  8269 raft_consensus.cc:3037] T 89f7cf12d85442749e7d5b259ebca43f P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:56:17.183696  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:17.227713  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:17.227834  8269 raft_consensus.cc:504] T 89f7cf12d85442749e7d5b259ebca43f P 436822a397664c54b8d1b9eb3bc6cff4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:56:17.256016  6200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89f7cf12d85442749e7d5b259ebca43f" candidate_uuid: "436822a397664c54b8d1b9eb3bc6cff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a2c90ee440b44b708bccc2e2d5385cc0"
I20240502 01:56:17.256103  6067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89f7cf12d85442749e7d5b259ebca43f" candidate_uuid: "436822a397664c54b8d1b9eb3bc6cff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "218bc8c1f9ce46059898fc738179adf9"
I20240502 01:56:17.256318  6200 raft_consensus.cc:3037] T 89f7cf12d85442749e7d5b259ebca43f P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:56:17.256345  6067 raft_consensus.cc:3037] T 89f7cf12d85442749e7d5b259ebca43f P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:56:17.259446  6200 raft_consensus.cc:2454] T 89f7cf12d85442749e7d5b259ebca43f P a2c90ee440b44b708bccc2e2d5385cc0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 436822a397664c54b8d1b9eb3bc6cff4 in term 2.
I20240502 01:56:17.259523  6067 raft_consensus.cc:2454] T 89f7cf12d85442749e7d5b259ebca43f P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 436822a397664c54b8d1b9eb3bc6cff4 in term 2.
I20240502 01:56:17.260519  8204 leader_election.cc:302] T 89f7cf12d85442749e7d5b259ebca43f P 436822a397664c54b8d1b9eb3bc6cff4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9, 436822a397664c54b8d1b9eb3bc6cff4, a2c90ee440b44b708bccc2e2d5385cc0; no voters: 
I20240502 01:56:17.261060 14016 raft_consensus.cc:2781] T 89f7cf12d85442749e7d5b259ebca43f P 436822a397664c54b8d1b9eb3bc6cff4 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:56:17.284651  6335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89f7cf12d85442749e7d5b259ebca43f" candidate_uuid: "436822a397664c54b8d1b9eb3bc6cff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "94ffd7aaf85b4e17881241b8396fa2f3"
I20240502 01:56:17.284956  6335 raft_consensus.cc:3032] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:56:17.285027  6335 raft_consensus.cc:729] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 94ffd7aaf85b4e17881241b8396fa2f3, State: Running, Role: LEADER
I20240502 01:56:17.285461  6335 consensus_queue.cc:260] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:56:17.285977  6335 raft_consensus.cc:3037] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:56:17.293902  6335 raft_consensus.cc:2454] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 436822a397664c54b8d1b9eb3bc6cff4 in term 2.
I20240502 01:56:17.302685  8269 leader_election.cc:288] T 89f7cf12d85442749e7d5b259ebca43f P 436822a397664c54b8d1b9eb3bc6cff4 [CANDIDATE]: Term 2 election: Requested vote from peers a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197), 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227), 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245)
I20240502 01:56:17.363116 14016 raft_consensus.cc:686] T 89f7cf12d85442749e7d5b259ebca43f P 436822a397664c54b8d1b9eb3bc6cff4 [term 2 LEADER]: Becoming Leader. State: Replica: 436822a397664c54b8d1b9eb3bc6cff4, State: Running, Role: LEADER
I20240502 01:56:17.364017 14016 consensus_queue.cc:237] T 89f7cf12d85442749e7d5b259ebca43f P 436822a397664c54b8d1b9eb3bc6cff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:56:17.391999  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:17.391912  5912 catalog_manager.cc:5484] T 89f7cf12d85442749e7d5b259ebca43f P 436822a397664c54b8d1b9eb3bc6cff4 reported cstate change: term changed from 1 to 2, leader changed from 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) to 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135). New cstate: current_term: 2 leader_uuid: "436822a397664c54b8d1b9eb3bc6cff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:17.439674  8406 consensus_queue.cc:237] T 73ed91df7ff04846991c95b30a86ea7e P 6d34e50fb5dd4a1db58013d4863b8992 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:56:17.451989  6200 raft_consensus.cc:1261] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 [term 2 FOLLOWER]: Refusing update from remote peer 6d34e50fb5dd4a1db58013d4863b8992: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 01:56:17.455075 13839 consensus_queue.cc:1035] T 73ed91df7ff04846991c95b30a86ea7e P 6d34e50fb5dd4a1db58013d4863b8992 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:56:17.477236  8269 raft_consensus.cc:1261] T 73ed91df7ff04846991c95b30a86ea7e P 436822a397664c54b8d1b9eb3bc6cff4 [term 2 FOLLOWER]: Refusing update from remote peer 6d34e50fb5dd4a1db58013d4863b8992: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 01:56:17.485563  6203 raft_consensus.cc:2932] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:17.485759 14110 consensus_queue.cc:1035] T 73ed91df7ff04846991c95b30a86ea7e P 6d34e50fb5dd4a1db58013d4863b8992 [LEADER]: Connected to new peer: Peer: permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:56:17.486482 14108 raft_consensus.cc:2932] T 73ed91df7ff04846991c95b30a86ea7e P 6d34e50fb5dd4a1db58013d4863b8992 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:17.493921  5912 catalog_manager.cc:5484] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 reported cstate change: config changed from index 7 to 8, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New cstate: current_term: 2 leader_uuid: "6d34e50fb5dd4a1db58013d4863b8992" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:17.511333  8269 raft_consensus.cc:2932] T 73ed91df7ff04846991c95b30a86ea7e P 436822a397664c54b8d1b9eb3bc6cff4 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:17.521251  6048 tablet_service.cc:1514] Processing DeleteTablet for tablet 73ed91df7ff04846991c95b30a86ea7e with delete_type TABLET_DATA_TOMBSTONED (TS 218bc8c1f9ce46059898fc738179adf9 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:50408
I20240502 01:56:17.592000 14118 tablet_replica.cc:330] T 73ed91df7ff04846991c95b30a86ea7e P 218bc8c1f9ce46059898fc738179adf9: stopping tablet replica
I20240502 01:56:17.592491 14118 raft_consensus.cc:2229] T 73ed91df7ff04846991c95b30a86ea7e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:17.600536 14118 raft_consensus.cc:2258] T 73ed91df7ff04846991c95b30a86ea7e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:17.607713  6048 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:17.609239  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:17.616698 14118 ts_tablet_manager.cc:1875] T 73ed91df7ff04846991c95b30a86ea7e P 218bc8c1f9ce46059898fc738179adf9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:56:17.629837  6447 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:17.633545  6316 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:56:17.638468  8385 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:56:17.671370  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:17.686020  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:17.688544  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:17.736522 14118 ts_tablet_manager.cc:1888] T 73ed91df7ff04846991c95b30a86ea7e P 218bc8c1f9ce46059898fc738179adf9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240502 01:56:17.737242 14118 log.cc:1196] T 73ed91df7ff04846991c95b30a86ea7e P 218bc8c1f9ce46059898fc738179adf9: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal/wals/73ed91df7ff04846991c95b30a86ea7e
I20240502 01:56:17.738315  5907 catalog_manager.cc:4830] TS 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227): tablet 73ed91df7ff04846991c95b30a86ea7e (table rebalance_test_table_6 [id=5606c3da105b4b86868c6c51ecdd39aa]) successfully deleted
I20240502 01:56:17.760100  6335 raft_consensus.cc:1261] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Refusing update from remote peer 436822a397664c54b8d1b9eb3bc6cff4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:56:17.761224 14016 consensus_queue.cc:1035] T 89f7cf12d85442749e7d5b259ebca43f P 436822a397664c54b8d1b9eb3bc6cff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:56:17.785122  6447 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:17.789009  6067 raft_consensus.cc:1261] T 89f7cf12d85442749e7d5b259ebca43f P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Refusing update from remote peer 436822a397664c54b8d1b9eb3bc6cff4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:56:17.789669 14016 consensus_queue.cc:1035] T 89f7cf12d85442749e7d5b259ebca43f P 436822a397664c54b8d1b9eb3bc6cff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:56:17.792903  6048 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:17.832367  6066 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d8a6db6ae11e4cb8881aee95bdb0e59d"
dest_uuid: "218bc8c1f9ce46059898fc738179adf9"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:58004
I20240502 01:56:17.832698  6066 raft_consensus.cc:595] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:56:17.876789  6200 raft_consensus.cc:1261] T 89f7cf12d85442749e7d5b259ebca43f P a2c90ee440b44b708bccc2e2d5385cc0 [term 2 FOLLOWER]: Refusing update from remote peer 436822a397664c54b8d1b9eb3bc6cff4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 01:56:17.877677 14087 consensus_queue.cc:1035] T 89f7cf12d85442749e7d5b259ebca43f P 436822a397664c54b8d1b9eb3bc6cff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:56:17.886220  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:17.902246  8385 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:56:17.978061  6316 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:56:17.981843  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:17.994581  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:18.075044 14135 raft_consensus.cc:979] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9: : Instructing follower 6d34e50fb5dd4a1db58013d4863b8992 to start an election
I20240502 01:56:18.075223 14135 raft_consensus.cc:1067] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9 [term 1 LEADER]: Signalling peer 6d34e50fb5dd4a1db58013d4863b8992 to start an election
I20240502 01:56:18.081229  8406 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d8a6db6ae11e4cb8881aee95bdb0e59d"
dest_uuid: "6d34e50fb5dd4a1db58013d4863b8992"
 from {username='slave'} at 127.25.109.129:45219
I20240502 01:56:18.081511  8406 raft_consensus.cc:482] T d8a6db6ae11e4cb8881aee95bdb0e59d P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:56:18.081631  8406 raft_consensus.cc:3037] T d8a6db6ae11e4cb8881aee95bdb0e59d P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:56:18.084805  8406 raft_consensus.cc:504] T d8a6db6ae11e4cb8881aee95bdb0e59d P 6d34e50fb5dd4a1db58013d4863b8992 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:56:18.086697  6335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a6db6ae11e4cb8881aee95bdb0e59d" candidate_uuid: "6d34e50fb5dd4a1db58013d4863b8992" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "94ffd7aaf85b4e17881241b8396fa2f3"
I20240502 01:56:18.086969  6335 raft_consensus.cc:3037] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:56:18.086890  6066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8a6db6ae11e4cb8881aee95bdb0e59d" candidate_uuid: "6d34e50fb5dd4a1db58013d4863b8992" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "218bc8c1f9ce46059898fc738179adf9"
I20240502 01:56:18.087105  6066 raft_consensus.cc:3032] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:56:18.087184  6066 raft_consensus.cc:729] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 218bc8c1f9ce46059898fc738179adf9, State: Running, Role: LEADER
I20240502 01:56:18.087548  6066 consensus_queue.cc:260] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:56:18.088038  6066 raft_consensus.cc:3037] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:56:18.089841  6335 raft_consensus.cc:2454] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d34e50fb5dd4a1db58013d4863b8992 in term 2.
I20240502 01:56:18.090504  8337 leader_election.cc:302] T d8a6db6ae11e4cb8881aee95bdb0e59d P 6d34e50fb5dd4a1db58013d4863b8992 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d34e50fb5dd4a1db58013d4863b8992, 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 
I20240502 01:56:18.090879 14108 raft_consensus.cc:2781] T d8a6db6ae11e4cb8881aee95bdb0e59d P 6d34e50fb5dd4a1db58013d4863b8992 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:56:18.090901  6066 raft_consensus.cc:2454] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d34e50fb5dd4a1db58013d4863b8992 in term 2.
I20240502 01:56:18.091543  8406 leader_election.cc:288] T d8a6db6ae11e4cb8881aee95bdb0e59d P 6d34e50fb5dd4a1db58013d4863b8992 [CANDIDATE]: Term 2 election: Requested vote from peers 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227), 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245)
I20240502 01:56:18.126639  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:18.154984 14108 raft_consensus.cc:686] T d8a6db6ae11e4cb8881aee95bdb0e59d P 6d34e50fb5dd4a1db58013d4863b8992 [term 2 LEADER]: Becoming Leader. State: Replica: 6d34e50fb5dd4a1db58013d4863b8992, State: Running, Role: LEADER
I20240502 01:56:18.155416 14108 consensus_queue.cc:237] T d8a6db6ae11e4cb8881aee95bdb0e59d P 6d34e50fb5dd4a1db58013d4863b8992 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:56:18.216969  5921 catalog_manager.cc:5484] T d8a6db6ae11e4cb8881aee95bdb0e59d P 6d34e50fb5dd4a1db58013d4863b8992 reported cstate change: term changed from 1 to 2, leader changed from 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) to 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136). New cstate: current_term: 2 leader_uuid: "6d34e50fb5dd4a1db58013d4863b8992" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:18.258162  8385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:18.291224  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:18.292235  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:18.297425  6447 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:18.319165  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:18.323753  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:18.355420  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:18.398608  6316 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:56:18.550945  6335 consensus_queue.cc:237] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:56:18.584265  8269 raft_consensus.cc:1261] T 2cefdfa8851147999c1425c606f36b1f P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:56:18.585750 13224 consensus_queue.cc:1035] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:56:18.598213  6465 raft_consensus.cc:1261] T 2cefdfa8851147999c1425c606f36b1f P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:56:18.598910 13224 consensus_queue.cc:1035] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:56:18.603267  8406 raft_consensus.cc:1261] T 2cefdfa8851147999c1425c606f36b1f P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:56:18.603942 13224 consensus_queue.cc:1035] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:56:18.624082  6335 raft_consensus.cc:1261] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Refusing update from remote peer 6d34e50fb5dd4a1db58013d4863b8992: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:56:18.624873 14108 consensus_queue.cc:1035] T d8a6db6ae11e4cb8881aee95bdb0e59d P 6d34e50fb5dd4a1db58013d4863b8992 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:56:18.680063 13224 raft_consensus.cc:2932] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:18.692799  8406 raft_consensus.cc:2932] T 2cefdfa8851147999c1425c606f36b1f P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:18.699368  5921 catalog_manager.cc:5484] T 2cefdfa8851147999c1425c606f36b1f P 6d34e50fb5dd4a1db58013d4863b8992 reported cstate change: config changed from index 5 to 6, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:18.702939  6465 raft_consensus.cc:2932] T 2cefdfa8851147999c1425c606f36b1f P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:18.704546  6180 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cefdfa8851147999c1425c606f36b1f with delete_type TABLET_DATA_TOMBSTONED (TS a2c90ee440b44b708bccc2e2d5385cc0 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41258
I20240502 01:56:18.716138 14299 tablet_replica.cc:330] T 2cefdfa8851147999c1425c606f36b1f P a2c90ee440b44b708bccc2e2d5385cc0: stopping tablet replica
I20240502 01:56:18.717870 14299 raft_consensus.cc:2229] T 2cefdfa8851147999c1425c606f36b1f P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:18.718267 14299 raft_consensus.cc:2258] T 2cefdfa8851147999c1425c606f36b1f P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:18.711719  8269 raft_consensus.cc:2932] T 2cefdfa8851147999c1425c606f36b1f P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:18.719193 14299 ts_tablet_manager.cc:1875] T 2cefdfa8851147999c1425c606f36b1f P a2c90ee440b44b708bccc2e2d5385cc0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:56:18.721390  6066 raft_consensus.cc:1261] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Refusing update from remote peer 6d34e50fb5dd4a1db58013d4863b8992: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:56:18.722074 14108 consensus_queue.cc:1035] T d8a6db6ae11e4cb8881aee95bdb0e59d P 6d34e50fb5dd4a1db58013d4863b8992 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:56:18.753764  6447 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:18.759601 14299 ts_tablet_manager.cc:1888] T 2cefdfa8851147999c1425c606f36b1f P a2c90ee440b44b708bccc2e2d5385cc0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:56:18.760643 14299 log.cc:1196] T 2cefdfa8851147999c1425c606f36b1f P a2c90ee440b44b708bccc2e2d5385cc0: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal/wals/2cefdfa8851147999c1425c606f36b1f
I20240502 01:56:18.763613  5908 catalog_manager.cc:4830] TS a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197): tablet 2cefdfa8851147999c1425c606f36b1f (table rebalance_test_table_9 [id=f406d4a60a5e40c485796d37ff4d76bc]) successfully deleted
I20240502 01:56:18.803408  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:18.829595  8385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:18.835619  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:18.854197  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:18.871766  6316 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:56:18.933307  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:18.981345  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:19.079028  6447 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:19.119194  8269 consensus_queue.cc:237] T 89f7cf12d85442749e7d5b259ebca43f P 436822a397664c54b8d1b9eb3bc6cff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:56:19.133929  6335 raft_consensus.cc:1261] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Refusing update from remote peer 436822a397664c54b8d1b9eb3bc6cff4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:56:19.134991 14016 consensus_queue.cc:1035] T 89f7cf12d85442749e7d5b259ebca43f P 436822a397664c54b8d1b9eb3bc6cff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:56:19.136946  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:19.138921  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:19.149034  6066 raft_consensus.cc:1261] T 89f7cf12d85442749e7d5b259ebca43f P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Refusing update from remote peer 436822a397664c54b8d1b9eb3bc6cff4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:56:19.156294 14016 consensus_queue.cc:1035] T 89f7cf12d85442749e7d5b259ebca43f P 436822a397664c54b8d1b9eb3bc6cff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:56:19.165731  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:19.178166  8385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:19.238927  6316 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:56:19.241793 14380 raft_consensus.cc:2932] T 89f7cf12d85442749e7d5b259ebca43f P 436822a397664c54b8d1b9eb3bc6cff4 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:19.241925  6335 raft_consensus.cc:2932] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:19.245231  6066 raft_consensus.cc:2932] T 89f7cf12d85442749e7d5b259ebca43f P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:19.250514  5921 catalog_manager.cc:5484] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3 reported cstate change: config changed from index 4 to 6, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New cstate: current_term: 2 leader_uuid: "436822a397664c54b8d1b9eb3bc6cff4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:19.302686  6180 tablet_service.cc:1514] Processing DeleteTablet for tablet 89f7cf12d85442749e7d5b259ebca43f with delete_type TABLET_DATA_TOMBSTONED (TS a2c90ee440b44b708bccc2e2d5385cc0 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41258
I20240502 01:56:19.303721 14401 tablet_replica.cc:330] T 89f7cf12d85442749e7d5b259ebca43f P a2c90ee440b44b708bccc2e2d5385cc0: stopping tablet replica
I20240502 01:56:19.304018 14401 raft_consensus.cc:2229] T 89f7cf12d85442749e7d5b259ebca43f P a2c90ee440b44b708bccc2e2d5385cc0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:19.304266 14401 raft_consensus.cc:2258] T 89f7cf12d85442749e7d5b259ebca43f P a2c90ee440b44b708bccc2e2d5385cc0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:19.305110 14401 ts_tablet_manager.cc:1875] T 89f7cf12d85442749e7d5b259ebca43f P a2c90ee440b44b708bccc2e2d5385cc0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:56:19.312881 14401 ts_tablet_manager.cc:1888] T 89f7cf12d85442749e7d5b259ebca43f P a2c90ee440b44b708bccc2e2d5385cc0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 01:56:19.313082 14401 log.cc:1196] T 89f7cf12d85442749e7d5b259ebca43f P a2c90ee440b44b708bccc2e2d5385cc0: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal/wals/89f7cf12d85442749e7d5b259ebca43f
I20240502 01:56:19.314249  5908 catalog_manager.cc:4830] TS a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197): tablet 89f7cf12d85442749e7d5b259ebca43f (table rebalance_test_table_1 [id=0053d72412224a79be83b7e007b51934]) successfully deleted
I20240502 01:56:19.322101  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:19.323801  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:19.586697  8406 consensus_queue.cc:237] T d8a6db6ae11e4cb8881aee95bdb0e59d P 6d34e50fb5dd4a1db58013d4863b8992 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:56:19.589318  6335 raft_consensus.cc:1261] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Refusing update from remote peer 6d34e50fb5dd4a1db58013d4863b8992: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:56:19.589973 14108 consensus_queue.cc:1035] T d8a6db6ae11e4cb8881aee95bdb0e59d P 6d34e50fb5dd4a1db58013d4863b8992 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:56:19.593119 14471 raft_consensus.cc:2932] T d8a6db6ae11e4cb8881aee95bdb0e59d P 6d34e50fb5dd4a1db58013d4863b8992 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:19.593729  6335 raft_consensus.cc:2932] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:19.610105  5917 catalog_manager.cc:5484] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 reported cstate change: config changed from index 5 to 7, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New cstate: current_term: 2 leader_uuid: "6d34e50fb5dd4a1db58013d4863b8992" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:19.614825  6048 tablet_service.cc:1514] Processing DeleteTablet for tablet d8a6db6ae11e4cb8881aee95bdb0e59d with delete_type TABLET_DATA_TOMBSTONED (TS 218bc8c1f9ce46059898fc738179adf9 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:50408
I20240502 01:56:19.627403 14478 tablet_replica.cc:330] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9: stopping tablet replica
I20240502 01:56:19.627641 14478 raft_consensus.cc:2229] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:19.627838 14478 raft_consensus.cc:2258] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:19.637706 14478 ts_tablet_manager.cc:1875] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:56:19.657235 14478 ts_tablet_manager.cc:1888] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 01:56:19.658263 14478 log.cc:1196] T d8a6db6ae11e4cb8881aee95bdb0e59d P 218bc8c1f9ce46059898fc738179adf9: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal/wals/d8a6db6ae11e4cb8881aee95bdb0e59d
I20240502 01:56:19.659360  5907 catalog_manager.cc:4830] TS 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227): tablet d8a6db6ae11e4cb8881aee95bdb0e59d (table rebalance_test_table_8 [id=fcde569cfdf747c2aaecac7d265b4b1f]) successfully deleted
I20240502 01:56:19.664505  8406 consensus_queue.cc:237] T d8a6db6ae11e4cb8881aee95bdb0e59d P 6d34e50fb5dd4a1db58013d4863b8992 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:56:19.667663  6335 raft_consensus.cc:1261] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Refusing update from remote peer 6d34e50fb5dd4a1db58013d4863b8992: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 01:56:19.668998 14471 consensus_queue.cc:1035] T d8a6db6ae11e4cb8881aee95bdb0e59d P 6d34e50fb5dd4a1db58013d4863b8992 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 01:56:19.670884  8337 consensus_peers.cc:485] T d8a6db6ae11e4cb8881aee95bdb0e59d P 6d34e50fb5dd4a1db58013d4863b8992 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d8a6db6ae11e4cb8881aee95bdb0e59d. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:56:19.675328 14471 raft_consensus.cc:2932] T d8a6db6ae11e4cb8881aee95bdb0e59d P 6d34e50fb5dd4a1db58013d4863b8992 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:19.681212  6335 raft_consensus.cc:2932] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:19.687999  5906 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet d8a6db6ae11e4cb8881aee95bdb0e59d with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 01:56:19.693646  5917 catalog_manager.cc:5484] T d8a6db6ae11e4cb8881aee95bdb0e59d P 6d34e50fb5dd4a1db58013d4863b8992 reported cstate change: config changed from index 7 to 8, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New cstate: current_term: 2 leader_uuid: "6d34e50fb5dd4a1db58013d4863b8992" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:19.728344  6200 consensus_queue.cc:237] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:56:19.734534  8269 raft_consensus.cc:1261] T 086509f1ee0a4a81b2f8194b50dbe159 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Refusing update from remote peer a2c90ee440b44b708bccc2e2d5385cc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:56:19.736177  8404 raft_consensus.cc:1261] T 086509f1ee0a4a81b2f8194b50dbe159 P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Refusing update from remote peer a2c90ee440b44b708bccc2e2d5385cc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:56:19.739656 13732 consensus_queue.cc:1035] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:56:19.740296 13732 consensus_queue.cc:1035] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:56:19.778102 14502 raft_consensus.cc:2932] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:19.780784  8404 raft_consensus.cc:2932] T 086509f1ee0a4a81b2f8194b50dbe159 P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:19.781605  8269 raft_consensus.cc:2932] T 086509f1ee0a4a81b2f8194b50dbe159 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:19.787773  5914 catalog_manager.cc:5484] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 reported cstate change: config changed from index 6 to 7, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New cstate: current_term: 1 leader_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:19.793808  6048 tablet_service.cc:1514] Processing DeleteTablet for tablet 086509f1ee0a4a81b2f8194b50dbe159 with delete_type TABLET_DATA_TOMBSTONED (TS 218bc8c1f9ce46059898fc738179adf9 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:50408
I20240502 01:56:19.794754 14478 tablet_replica.cc:330] T 086509f1ee0a4a81b2f8194b50dbe159 P 218bc8c1f9ce46059898fc738179adf9: stopping tablet replica
I20240502 01:56:19.795009 14478 raft_consensus.cc:2229] T 086509f1ee0a4a81b2f8194b50dbe159 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:19.795261 14478 raft_consensus.cc:2258] T 086509f1ee0a4a81b2f8194b50dbe159 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:19.796051 14478 ts_tablet_manager.cc:1875] T 086509f1ee0a4a81b2f8194b50dbe159 P 218bc8c1f9ce46059898fc738179adf9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:56:19.820015 14478 ts_tablet_manager.cc:1888] T 086509f1ee0a4a81b2f8194b50dbe159 P 218bc8c1f9ce46059898fc738179adf9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:56:19.827020 14478 log.cc:1196] T 086509f1ee0a4a81b2f8194b50dbe159 P 218bc8c1f9ce46059898fc738179adf9: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal/wals/086509f1ee0a4a81b2f8194b50dbe159
I20240502 01:56:19.828109  5907 catalog_manager.cc:4830] TS 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227): tablet 086509f1ee0a4a81b2f8194b50dbe159 (table rebalance_test_table_0 [id=ef6f7a2e933049e5982deec12e9a670a]) successfully deleted
I20240502 01:56:19.883147  6447 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:19.890588  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:19.908331  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:19.932209  8385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:19.949739  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:20.033659  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:20.099664  6316 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:56:20.101271  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:20.219576 14583 ts_tablet_manager.cc:902] T d8a6db6ae11e4cb8881aee95bdb0e59d P 9db17f84d37944098cbeb9e0d8456efb: Initiating tablet copy from peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359)
I20240502 01:56:20.238492  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:20.244040  6447 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:20.259148  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:20.284845  8385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:20.325904  6316 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:56:20.329324  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:20.330080  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:20.341508  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:20.347854  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:20.368093  6447 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:20.422300  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:20.440982  6316 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:56:20.444610  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:20.466015  8385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:20.476509 14583 tablet_copy_client.cc:308] T d8a6db6ae11e4cb8881aee95bdb0e59d P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.136:44359
I20240502 01:56:20.480607  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:20.526808  8426 tablet_copy_service.cc:140] P 6d34e50fb5dd4a1db58013d4863b8992: Received BeginTabletCopySession request for tablet d8a6db6ae11e4cb8881aee95bdb0e59d from peer 9db17f84d37944098cbeb9e0d8456efb ({username='slave'} at 127.25.109.134:47537)
I20240502 01:56:20.527035  8426 tablet_copy_service.cc:161] P 6d34e50fb5dd4a1db58013d4863b8992: Beginning new tablet copy session on tablet d8a6db6ae11e4cb8881aee95bdb0e59d from peer 9db17f84d37944098cbeb9e0d8456efb at {username='slave'} at 127.25.109.134:47537: session id = 9db17f84d37944098cbeb9e0d8456efb-d8a6db6ae11e4cb8881aee95bdb0e59d
I20240502 01:56:20.530715  8426 tablet_copy_source_session.cc:205] T d8a6db6ae11e4cb8881aee95bdb0e59d P 6d34e50fb5dd4a1db58013d4863b8992: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:56:20.532697 14583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8a6db6ae11e4cb8881aee95bdb0e59d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:56:20.539925 14583 tablet_copy_client.cc:783] T d8a6db6ae11e4cb8881aee95bdb0e59d P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Starting download of 0 data blocks...
I20240502 01:56:20.540252 14583 tablet_copy_client.cc:647] T d8a6db6ae11e4cb8881aee95bdb0e59d P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Starting download of 1 WAL segments...
I20240502 01:56:20.542500 14583 tablet_copy_client.cc:523] T d8a6db6ae11e4cb8881aee95bdb0e59d P 9db17f84d37944098cbeb9e0d8456efb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:56:20.546854 14583 tablet_bootstrap.cc:492] T d8a6db6ae11e4cb8881aee95bdb0e59d P 9db17f84d37944098cbeb9e0d8456efb: Bootstrap starting.
I20240502 01:56:20.576752  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:20.651084  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:20.677739  6447 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:20.685469  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:20.702324  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:20.723377 14583 tablet_bootstrap.cc:492] T d8a6db6ae11e4cb8881aee95bdb0e59d P 9db17f84d37944098cbeb9e0d8456efb: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:56:20.723888 14583 tablet_bootstrap.cc:492] T d8a6db6ae11e4cb8881aee95bdb0e59d P 9db17f84d37944098cbeb9e0d8456efb: Bootstrap complete.
I20240502 01:56:20.726047 14583 ts_tablet_manager.cc:1367] T d8a6db6ae11e4cb8881aee95bdb0e59d P 9db17f84d37944098cbeb9e0d8456efb: Time spent bootstrapping tablet: real 0.179s	user 0.050s	sys 0.009s
I20240502 01:56:20.726797 14583 raft_consensus.cc:348] T d8a6db6ae11e4cb8881aee95bdb0e59d P 9db17f84d37944098cbeb9e0d8456efb [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:56:20.727020 14583 raft_consensus.cc:729] T d8a6db6ae11e4cb8881aee95bdb0e59d P 9db17f84d37944098cbeb9e0d8456efb [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9db17f84d37944098cbeb9e0d8456efb, State: Initialized, Role: FOLLOWER
I20240502 01:56:20.727236 14583 consensus_queue.cc:260] T d8a6db6ae11e4cb8881aee95bdb0e59d P 9db17f84d37944098cbeb9e0d8456efb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:56:20.756505  8385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:20.763470  6316 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:56:20.801198  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:20.801671  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:20.812775 14583 ts_tablet_manager.cc:1398] T d8a6db6ae11e4cb8881aee95bdb0e59d P 9db17f84d37944098cbeb9e0d8456efb: Time spent starting tablet: real 0.087s	user 0.006s	sys 0.024s
I20240502 01:56:20.824149  8426 tablet_copy_service.cc:342] P 6d34e50fb5dd4a1db58013d4863b8992: Request end of tablet copy session 9db17f84d37944098cbeb9e0d8456efb-d8a6db6ae11e4cb8881aee95bdb0e59d received from {username='slave'} at 127.25.109.134:47537
I20240502 01:56:20.824352  8426 tablet_copy_service.cc:433] P 6d34e50fb5dd4a1db58013d4863b8992: ending tablet copy session 9db17f84d37944098cbeb9e0d8456efb-d8a6db6ae11e4cb8881aee95bdb0e59d on tablet d8a6db6ae11e4cb8881aee95bdb0e59d with peer 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:56:20.920089  6465 consensus_queue.cc:237] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:56:20.942301  8404 raft_consensus.cc:1261] T 4eb91f5df851459b8ecf09cbe34fefa5 P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:56:20.943125 14548 consensus_queue.cc:1035] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:56:20.956226  8134 raft_consensus.cc:1261] T 4eb91f5df851459b8ecf09cbe34fefa5 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:56:20.956801 14548 raft_consensus.cc:2932] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:20.992594 14681 consensus_queue.cc:1035] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:56:21.004681  8134 raft_consensus.cc:2932] T 4eb91f5df851459b8ecf09cbe34fefa5 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:21.013959  5914 catalog_manager.cc:5484] T 4eb91f5df851459b8ecf09cbe34fefa5 P 9db17f84d37944098cbeb9e0d8456efb reported cstate change: config changed from index 6 to 7, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New cstate: current_term: 1 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:21.035593  8404 raft_consensus.cc:2932] T 4eb91f5df851459b8ecf09cbe34fefa5 P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:21.067934  6180 tablet_service.cc:1514] Processing DeleteTablet for tablet 4eb91f5df851459b8ecf09cbe34fefa5 with delete_type TABLET_DATA_TOMBSTONED (TS a2c90ee440b44b708bccc2e2d5385cc0 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:41258
I20240502 01:56:21.116406 14699 tablet_replica.cc:330] T 4eb91f5df851459b8ecf09cbe34fefa5 P a2c90ee440b44b708bccc2e2d5385cc0: stopping tablet replica
I20240502 01:56:21.118891 14699 raft_consensus.cc:2229] T 4eb91f5df851459b8ecf09cbe34fefa5 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:21.119215 14699 raft_consensus.cc:2258] T 4eb91f5df851459b8ecf09cbe34fefa5 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:21.122567 14699 ts_tablet_manager.cc:1875] T 4eb91f5df851459b8ecf09cbe34fefa5 P a2c90ee440b44b708bccc2e2d5385cc0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:56:21.136611 14699 ts_tablet_manager.cc:1888] T 4eb91f5df851459b8ecf09cbe34fefa5 P a2c90ee440b44b708bccc2e2d5385cc0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:56:21.136790 14699 log.cc:1196] T 4eb91f5df851459b8ecf09cbe34fefa5 P a2c90ee440b44b708bccc2e2d5385cc0: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal/wals/4eb91f5df851459b8ecf09cbe34fefa5
I20240502 01:56:21.137609  5908 catalog_manager.cc:4830] TS a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197): tablet 4eb91f5df851459b8ecf09cbe34fefa5 (table rebalance_test_table_1 [id=0053d72412224a79be83b7e007b51934]) successfully deleted
I20240502 01:56:21.154467  6465 consensus_queue.cc:237] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 01:56:21.162448  6200 consensus_queue.cc:237] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:56:21.164767  8001 raft_consensus.cc:1261] T 02b106ec57b94e44a29947a80a030a23 P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:56:21.168576  6203 raft_consensus.cc:1261] T 02b106ec57b94e44a29947a80a030a23 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:56:21.174863 14680 consensus_queue.cc:1035] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:56:21.175653 14680 consensus_queue.cc:1035] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:56:21.177919  8404 raft_consensus.cc:1261] T d7922fd756344d14a336ff0c54afdf73 P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Refusing update from remote peer a2c90ee440b44b708bccc2e2d5385cc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:56:21.179168 14502 consensus_queue.cc:1035] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:56:21.192797 14181 raft_consensus.cc:2932] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:56:21.196734  6200 raft_consensus.cc:2932] T 02b106ec57b94e44a29947a80a030a23 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:56:21.206236  5921 catalog_manager.cc:5484] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 reported cstate change: config changed from index 4 to 5, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New cstate: current_term: 1 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:56:21.211675  6048 tablet_service.cc:1514] Processing DeleteTablet for tablet 02b106ec57b94e44a29947a80a030a23 with delete_type TABLET_DATA_TOMBSTONED (TS 218bc8c1f9ce46059898fc738179adf9 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:50408
I20240502 01:56:21.220571  8001 raft_consensus.cc:2932] T 02b106ec57b94e44a29947a80a030a23 P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
W20240502 01:56:21.233712  6135 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.136:44359, user_credentials={real_user=slave}} blocked reactor thread for 55051.6us
I20240502 01:56:21.259667 14735 raft_consensus.cc:2932] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:21.260763  8404 raft_consensus.cc:2932] T d7922fd756344d14a336ff0c54afdf73 P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:21.268074  5921 catalog_manager.cc:5484] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 reported cstate change: config changed from index 5 to 6, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New cstate: current_term: 1 leader_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:21.273097  6447 tablet_service.cc:1514] Processing DeleteTablet for tablet d7922fd756344d14a336ff0c54afdf73 with delete_type TABLET_DATA_TOMBSTONED (TS 07cdb213c77c4ffda1f999b66b725399 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56564
I20240502 01:56:21.274395  6203 consensus_queue.cc:237] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
W20240502 01:56:21.277588  6136 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7922fd756344d14a336ff0c54afdf73. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:56:21.295251 14751 tablet_replica.cc:330] T d7922fd756344d14a336ff0c54afdf73 P 07cdb213c77c4ffda1f999b66b725399: stopping tablet replica
I20240502 01:56:21.296031 14751 raft_consensus.cc:2229] T d7922fd756344d14a336ff0c54afdf73 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:21.296995 14751 raft_consensus.cc:2258] T d7922fd756344d14a336ff0c54afdf73 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:21.299436  8135 raft_consensus.cc:1203] T d8a6db6ae11e4cb8881aee95bdb0e59d P 9db17f84d37944098cbeb9e0d8456efb [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8]   Dedup: 2.8->[]
I20240502 01:56:21.305516  8406 raft_consensus.cc:1261] T d7922fd756344d14a336ff0c54afdf73 P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Refusing update from remote peer a2c90ee440b44b708bccc2e2d5385cc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:56:21.306456 14502 consensus_queue.cc:1035] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:56:21.311465 14502 raft_consensus.cc:2932] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:21.316466  8406 raft_consensus.cc:2932] T d7922fd756344d14a336ff0c54afdf73 P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:21.324517  5921 catalog_manager.cc:5484] T d7922fd756344d14a336ff0c54afdf73 P 6d34e50fb5dd4a1db58013d4863b8992 reported cstate change: config changed from index 6 to 7, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) added. New cstate: current_term: 1 leader_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:21.326661  5908 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet d7922fd756344d14a336ff0c54afdf73 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 01:56:21.325585  6465 consensus_queue.cc:237] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:56:21.335924 14751 ts_tablet_manager.cc:1875] T d7922fd756344d14a336ff0c54afdf73 P 07cdb213c77c4ffda1f999b66b725399: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:56:21.336156  8135 raft_consensus.cc:1261] T 6984d638991442e3adaed1b24bee95e9 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:56:21.336768 14181 consensus_queue.cc:1035] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:56:21.344363  6066 raft_consensus.cc:1261] T 6984d638991442e3adaed1b24bee95e9 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:56:21.346966 14680 consensus_queue.cc:1035] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:56:21.361126  8001 raft_consensus.cc:1261] T 6984d638991442e3adaed1b24bee95e9 P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:56:21.361986 14548 consensus_queue.cc:1035] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:56:21.379566  8404 raft_consensus.cc:1261] T 6984d638991442e3adaed1b24bee95e9 P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:56:21.380409 14548 consensus_queue.cc:1035] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:56:21.392565 14741 tablet_replica.cc:330] T 02b106ec57b94e44a29947a80a030a23 P 218bc8c1f9ce46059898fc738179adf9: stopping tablet replica
I20240502 01:56:21.395207 14741 raft_consensus.cc:2229] T 02b106ec57b94e44a29947a80a030a23 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:21.395515 14741 raft_consensus.cc:2258] T 02b106ec57b94e44a29947a80a030a23 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:21.396351 14741 ts_tablet_manager.cc:1875] T 02b106ec57b94e44a29947a80a030a23 P 218bc8c1f9ce46059898fc738179adf9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 01:56:21.400884  6271 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.130:38197, user_credentials={real_user=slave}} blocked reactor thread for 54207.8us
W20240502 01:56:21.405979  6269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.136:44359, user_credentials={real_user=slave}} blocked reactor thread for 46037.2us
I20240502 01:56:21.409457 14741 ts_tablet_manager.cc:1888] T 02b106ec57b94e44a29947a80a030a23 P 218bc8c1f9ce46059898fc738179adf9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:56:21.409612 14741 log.cc:1196] T 02b106ec57b94e44a29947a80a030a23 P 218bc8c1f9ce46059898fc738179adf9: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal/wals/02b106ec57b94e44a29947a80a030a23
I20240502 01:56:21.409237 14680 raft_consensus.cc:2932] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:21.412387  5907 catalog_manager.cc:4830] TS 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227): tablet 02b106ec57b94e44a29947a80a030a23 (table rebalance_test_table_2 [id=ef7ebedfbbff4054b074bf7a7622b632]) successfully deleted
I20240502 01:56:21.417181  8406 raft_consensus.cc:2932] T 6984d638991442e3adaed1b24bee95e9 P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:21.419518  8001 raft_consensus.cc:2932] T 6984d638991442e3adaed1b24bee95e9 P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:21.421245  5914 catalog_manager.cc:5484] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 reported cstate change: config changed from index 6 to 7, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New cstate: current_term: 1 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:21.433431  6316 tablet_service.cc:1514] Processing DeleteTablet for tablet 6984d638991442e3adaed1b24bee95e9 with delete_type TABLET_DATA_TOMBSTONED (TS 94ffd7aaf85b4e17881241b8396fa2f3 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:41134
I20240502 01:56:21.442653  8134 raft_consensus.cc:2932] T 6984d638991442e3adaed1b24bee95e9 P 9db17f84d37944098cbeb9e0d8456efb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:21.445129 14751 ts_tablet_manager.cc:1888] T d7922fd756344d14a336ff0c54afdf73 P 07cdb213c77c4ffda1f999b66b725399: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:56:21.440932  6066 raft_consensus.cc:2932] T 6984d638991442e3adaed1b24bee95e9 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:21.455235 14786 tablet_replica.cc:330] T 6984d638991442e3adaed1b24bee95e9 P 94ffd7aaf85b4e17881241b8396fa2f3: stopping tablet replica
I20240502 01:56:21.457111 14786 raft_consensus.cc:2229] T 6984d638991442e3adaed1b24bee95e9 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:21.460331 14751 log.cc:1196] T d7922fd756344d14a336ff0c54afdf73 P 07cdb213c77c4ffda1f999b66b725399: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal/wals/d7922fd756344d14a336ff0c54afdf73
I20240502 01:56:21.457417 14786 raft_consensus.cc:2258] T 6984d638991442e3adaed1b24bee95e9 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:21.461230 14786 ts_tablet_manager.cc:1875] T 6984d638991442e3adaed1b24bee95e9 P 94ffd7aaf85b4e17881241b8396fa2f3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:56:21.461972  5905 catalog_manager.cc:4830] TS 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867): tablet d7922fd756344d14a336ff0c54afdf73 (table rebalance_test_table_5 [id=c02dbeef0b7241e78a12e4bd7ef79e47]) successfully deleted
I20240502 01:56:21.577956 14786 ts_tablet_manager.cc:1888] T 6984d638991442e3adaed1b24bee95e9 P 94ffd7aaf85b4e17881241b8396fa2f3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:56:21.587317  6447 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:21.632347 14786 log.cc:1196] T 6984d638991442e3adaed1b24bee95e9 P 94ffd7aaf85b4e17881241b8396fa2f3: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/wals/6984d638991442e3adaed1b24bee95e9
I20240502 01:56:21.633479  5907 catalog_manager.cc:4830] TS 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245): tablet 6984d638991442e3adaed1b24bee95e9 (table rebalance_test_table_3 [id=63a163be603345428cc84c86b96467f0]) successfully deleted
I20240502 01:56:21.634085  8385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:21.636353  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:21.656975  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:56:21.673182  5998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.135:40543, user_credentials={real_user=slave}} blocked reactor thread for 61432.4us
I20240502 01:56:21.683444  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:21.720820  6316 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:56:21.750231  8115 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:21.782384  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:21.835112 14846 ts_tablet_manager.cc:902] T d7922fd756344d14a336ff0c54afdf73 P 9db17f84d37944098cbeb9e0d8456efb: Initiating tablet copy from peer a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197)
F20240502 01:56:21.837697 14846 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0cecd1b14e474dee825fae040ce1b31e.queue_time_us, tablet-download-pool-0cecd1b14e474dee825fae040ce1b31e.queue_time_us) (tablet-download-pool-0cecd1b14e474dee825fae040ce1b31e.run_cpu_time_us, tablet-download-pool-0cecd1b14e474dee825fae040ce1b31e.run_cpu_time_us) (tablet-download-pool-0cecd1b14e474dee825fae040ce1b31e.run_wall_time_us, tablet-download-pool-0cecd1b14e474dee825fae040ce1b31e.run_wall_time_us) (tablet-download-pool-265a9645e1f141cb88c786b9aef8f90e.queue_time_us, tablet-download-pool-265a9645e1f141cb88c786b9aef8f90e.queue_time_us) (tablet-download-pool-265a9645e1f141cb88c786b9aef8f90e.run_cpu_time_us, tablet-download-pool-265a9645e1f141cb88c786b9aef8f90e.run_cpu_time_us) (tablet-download-pool-265a9645e1f141cb88c786b9aef8f90e.run_wall_time_us, tablet-download-pool-265a9645e1f141cb88c786b9aef8f90e.run_wall_time_us) (tablet-download-pool-4eb91f5df851459b8ecf09cbe34fefa5.queue_time_us, tablet-download-pool-4eb91f5df851459b8ecf09cbe34fefa5.queue_time_us) (tablet-download-pool-4eb91f5df851459b8ecf09cbe34fefa5.run_cpu_time_us, tablet-download-pool-4eb91f5df851459b8ecf09cbe34fefa5.run_cpu_time_us) (tablet-download-pool-4eb91f5df851459b8ecf09cbe34fefa5.run_wall_time_us, tablet-download-pool-4eb91f5df851459b8ecf09cbe34fefa5.run_wall_time_us) (tablet-download-pool-4f4088f3523f4cd0afdf0de0f447c7af.queue_time_us, tablet-download-pool-4f4088f3523f4cd0afdf0de0f447c7af.queue_time_us) (tablet-download-pool-4f4088f3523f4cd0afdf0de0f447c7af.run_cpu_time_us, tablet-download-pool-4f4088f3523f4cd0afdf0de0f447c7af.run_cpu_time_us) (tablet-download-pool-4f4088f3523f4cd0afdf0de0f447c7af.run_wall_time_us, tablet-download-pool-4f4088f3523f4cd0afdf0de0f447c7af.run_wall_time_us) (tablet-download-pool-5be45a68cff24a75a03b2fcd4b7851f9.queue_time_us, tablet-download-pool-5be45a68cff24a75a03b2fcd4b7851f9.queue_time_us) (tablet-download-pool-5be45a68cff24a75a03b2fcd4b7851f9.run_cpu_time_us, tablet-download-pool-5be45a68cff24a75a03b2fcd4b7851f9.run_cpu_time_us) (tablet-download-pool-5be45a68cff24a75a03b2fcd4b7851f9.run_wall_time_us, tablet-download-pool-5be45a68cff24a75a03b2fcd4b7851f9.run_wall_time_us) (tablet-download-pool-6984d638991442e3adaed1b24bee95e9.queue_time_us, tablet-download-pool-6984d638991442e3adaed1b24bee95e9.queue_time_us) (tablet-download-pool-6984d638991442e3adaed1b24bee95e9.run_cpu_time_us, tablet-download-pool-6984d638991442e3adaed1b24bee95e9.run_cpu_time_us) (tablet-download-pool-6984d638991442e3adaed1b24bee95e9.run_wall_time_us, tablet-download-pool-6984d638991442e3adaed1b24bee95e9.run_wall_time_us) (tablet-download-pool-7f2519b0c6404799ad84d120b45a7ea4.queue_time_us, tablet-download-pool-7f2519b0c6404799ad84d120b45a7ea4.queue_time_us) (tablet-download-pool-7f2519b0c6404799ad84d120b45a7ea4.run_cpu_time_us, tablet-download-pool-7f2519b0c6404799ad84d120b45a7ea4.run_cpu_time_us) (tablet-download-pool-7f2519b0c6404799ad84d120b45a7ea4.run_wall_time_us, tablet-download-pool-7f2519b0c6404799ad84d120b45a7ea4.run_wall_time_us) (tablet-download-pool-899963a8260a42c4b73fb80eda2ecaa9.queue_time_us, tablet-download-pool-899963a8260a42c4b73fb80eda2ecaa9.queue_time_us) (tablet-download-pool-899963a8260a42c4b73fb80eda2ecaa9.run_cpu_time_us, tablet-download-pool-899963a8260a42c4b73fb80eda2ecaa9.run_cpu_time_us) (tablet-download-pool-899963a8260a42c4b73fb80eda2ecaa9.run_wall_time_us, tablet-download-pool-899963a8260a42c4b73fb80eda2ecaa9.run_wall_time_us) (tablet-download-pool-99e74a49b42745888058db67049dc682.queue_time_us, tablet-download-pool-99e74a49b42745888058db67049dc682.queue_time_us) (tablet-download-pool-99e74a49b42745888058db67049dc682.run_cpu_time_us, tablet-download-pool-99e74a49b42745888058db67049dc682.run_cpu_time_us) (tablet-download-pool-99e74a49b42745888058db67049dc682.run_wall_time_us, tablet-download-pool-99e74a49b42745888058db67049dc682.run_wall_time_us) (tablet-download-pool-afa5561f8c0e4c0fa9bbbd5da61348c7.queue_time_us, tablet-download-pool-afa5561f8c0e4c0fa9bbbd5da61348c7.queue_time_us) (tablet-download-pool-afa5561f8c0e4c0fa9bbbd5da61348c7.run_cpu_time_us, tablet-download-pool-afa5561f8c0e4c0fa9bbbd5da61348c7.run_cpu_time_us) (tablet-download-pool-afa5561f8c0e4c0fa9bbbd5da61348c7.run_wall_time_us, tablet-download-pool-afa5561f8c0e4c0fa9bbbd5da61348c7.run_wall_time_us) (tablet-download-pool-c58d6b2401c047cd9af56a66c2e00105.queue_time_us, tablet-download-pool-c58d6b2401c047cd9af56a66c2e00105.queue_time_us) (tablet-download-pool-c58d6b2401c047cd9af56a66c2e00105.run_cpu_time_us, tablet-download-pool-c58d6b2401c047cd9af56a66c2e00105.run_cpu_time_us) (tablet-download-pool-c58d6b2401c047cd9af56a66c2e00105.run_wall_time_us, tablet-download-pool-c58d6b2401c047cd9af56a66c2e00105.run_wall_time_us) (tablet-download-pool-ca458907378e42d48adaa0ee647db4cd.queue_time_us, tablet-download-pool-ca458907378e42d48adaa0ee647db4cd.queue_time_us) (tablet-download-pool-ca458907378e42d48adaa0ee647db4cd.run_cpu_time_us, tablet-download-pool-ca458907378e42d48adaa0ee647db4cd.run_cpu_time_us) (tablet-download-pool-ca458907378e42d48adaa0ee647db4cd.run_wall_time_us, tablet-download-pool-ca458907378e42d48adaa0ee647db4cd.run_wall_time_us) (tablet-download-pool-d7922fd756344d14a336ff0c54afdf73.queue_time_us, tablet-download-pool-d7922fd756344d14a336ff0c54afdf73.queue_time_us) (tablet-download-pool-d8a6db6ae11e4cb8881aee95bdb0e59d.queue_time_us, tablet-download-pool-d8a6db6ae11e4cb8881aee95bdb0e59d.queue_time_us) (tablet-download-pool-d8a6db6ae11e4cb8881aee95bdb0e59d.run_cpu_time_us, tablet-download-pool-d8a6db6ae11e4cb8881aee95bdb0e59d.run_cpu_time_us) (tablet-download-pool-d8a6db6ae11e4cb8881aee95bdb0e59d.run_wall_time_us, tablet-download-pool-d8a6db6ae11e4cb8881aee95bdb0e59d.run_wall_time_us) (tablet-download-pool-f309b40931dd444091869119eab33bbc.queue_time_us, tablet-download-pool-f309b40931dd444091869119eab33bbc.queue_time_us) (tablet-download-pool-f309b40931dd444091869119eab33bbc.run_cpu_time_us, tablet-download-pool-f309b40931dd444091869119eab33bbc.run_cpu_time_us) (tablet-download-pool-f309b40931dd444091869119eab33bbc.run_wall_time_us, tablet-download-pool-f309b40931dd444091869119eab33bbc.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
    @     0x7fad23ac7dcd  google::LogMessage::Fail() at ??:0
    @     0x7fad23acbb93  google::LogMessage::SendToLog() at ??:0
    @     0x7fad23ac77cc  google::LogMessage::Flush() at ??:0
    @     0x7fad23ac8f59  google::LogMessageFatal::~LogMessageFatal() at ??:0
    @     0x7fad26808742  kudu::TraceMetrics::InternName() at ??:0
    @     0x7fad267d4f8b  kudu::ThreadPool::ThreadPool() at ??:0
    @     0x7fad267cecd6  kudu::ThreadPoolBuilder::Build() at ??:0
    @     0x7fad35c218de  kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
    @     0x7fad35c39daf  kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
    @     0x7fad35d64480  kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
    @     0x7fad35d9ca8d  kudu::tserver::TabletCopyRunnable::Run() at ??:0
    @     0x7fad35d7f981  kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
    @     0x7fad35d7f70e  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa60bcd  std::function<>::operator()() at ??:?
    @     0x7fad267d8b75  kudu::ThreadPool::DispatchThread() at ??:0
    @     0x7fad267dc19c  kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
    @     0x7fad267dc019  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa60bcd  std::function<>::operator()() at ??:?
    @     0x7fad267b701e  kudu::Thread::SuperviseThread() at ??:0
    @     0x7fad29b3c6db  start_thread at ??:0
    @     0x7fad2297a71f  clone at ??:0
I20240502 01:56:22.200294  6465 consensus_queue.cc:237] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:56:22.247947  6203 raft_consensus.cc:1261] T f309b40931dd444091869119eab33bbc P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:56:22.248842 14680 consensus_queue.cc:1035] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:56:22.262756  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:22.317749 14960 raft_consensus.cc:2932] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:22.327194  5917 catalog_manager.cc:5484] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 reported cstate change: config changed from index 4 to 5, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New cstate: current_term: 1 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:22.328635  6200 raft_consensus.cc:2932] T f309b40931dd444091869119eab33bbc P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } }
I20240502 01:56:22.370882  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:22.372867  6447 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:22.383421  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:22.396929  6316 tablet_service.cc:1514] Processing DeleteTablet for tablet f309b40931dd444091869119eab33bbc with delete_type TABLET_DATA_TOMBSTONED (TS 94ffd7aaf85b4e17881241b8396fa2f3 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41134
I20240502 01:56:22.421197  6314 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:56:22.442786 14975 tablet_replica.cc:330] T f309b40931dd444091869119eab33bbc P 94ffd7aaf85b4e17881241b8396fa2f3: stopping tablet replica
I20240502 01:56:22.443075 14975 raft_consensus.cc:2229] T f309b40931dd444091869119eab33bbc P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:22.443289 14975 raft_consensus.cc:2258] T f309b40931dd444091869119eab33bbc P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:22.444144 14975 ts_tablet_manager.cc:1875] T f309b40931dd444091869119eab33bbc P 94ffd7aaf85b4e17881241b8396fa2f3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:56:22.458211  8385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:22.506209  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:22.577250  6447 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:22.603614 14975 ts_tablet_manager.cc:1888] T f309b40931dd444091869119eab33bbc P 94ffd7aaf85b4e17881241b8396fa2f3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 01:56:22.603739 14975 log.cc:1196] T f309b40931dd444091869119eab33bbc P 94ffd7aaf85b4e17881241b8396fa2f3: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/wals/f309b40931dd444091869119eab33bbc
I20240502 01:56:22.604470  5907 catalog_manager.cc:4830] TS 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245): tablet f309b40931dd444091869119eab33bbc (table rebalance_test_table_9 [id=f406d4a60a5e40c485796d37ff4d76bc]) successfully deleted
I20240502 01:56:22.606560  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:22.630533  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:22.631757  8385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:22.644114  6316 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:56:22.680193  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:22.692734  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:56:22.725229  8337 connection.cc:535] client connection to 127.25.109.134:42517 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:56:22.725476  8337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:56:22.736873  6270 connection.cc:535] client connection to 127.25.109.134:42517 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:56:22.737120  6270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:56:22.744011  6000 connection.cc:535] client connection to 127.25.109.134:42517 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:56:22.744297  6000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:56:22.757218  6136 connection.cc:535] client connection to 127.25.109.134:42517 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:56:22.757524  6136 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:56:22.759423  6402 connection.cc:535] client connection to 127.25.109.134:42517 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:56:22.759696  6402 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:56:22.783107  8337 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.134:42517, user_credentials={real_user=slave}} blocked reactor thread for 57687.9us
W20240502 01:56:22.785574 15023 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 01:56:22.744052 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.134:42517 (local address 127.25.109.131:40461)
0502 01:56:22.780402 (+ 36350us) negotiation.cc:107] Waiting for socket to connect
0502 01:56:22.780545 (+   143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":36227,"thread_start_us":35901,"threads_started":1}
W20240502 01:56:22.789804  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:22.789944  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:22.790005  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:22.790062  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:22.790117  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:22.791911  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:22.792047  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:22.794057  8337 consensus_peers.cc:485] T d8a6db6ae11e4cb8881aee95bdb0e59d P 6d34e50fb5dd4a1db58013d4863b8992 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:22.794178  8337 consensus_peers.cc:485] T 265a9645e1f141cb88c786b9aef8f90e P 6d34e50fb5dd4a1db58013d4863b8992 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:22.804095  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:56:22.804242  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:22.838538  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:22.838690  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:22.838821  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:22.838915  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:22.840250  5999 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.136:44359, user_credentials={real_user=slave}} blocked reactor thread for 45686.4us
I20240502 01:56:22.852134  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:22.862911  8385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:22.871444  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:22.885010  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:22.886649  6447 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:22.911934  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:22.919935  6316 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 01:56:23.333678  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:23.432983  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:23.435670  6447 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:23.497624  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:23.525293  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:23.532188  8385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:23.576802  6316 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:56:23.613121  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:23.728536  6447 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:23.747721  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:23.761415  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:56:23.768987  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:23.792610  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:23.857919  8385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:23.895057  6316 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:56:23.920753  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:24.210584  6447 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:24.212813  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:56:24.230537  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:24.252494  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:24.276235  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:24.311247  8385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:24.363893  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:24.377619  6335 consensus_queue.cc:237] T ac3325a06d8841718158e1f5f181a398 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 01:56:24.393577  7998 raft_consensus.cc:1261] T ac3325a06d8841718158e1f5f181a398 P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:56:24.394232 15125 consensus_queue.cc:1035] T ac3325a06d8841718158e1f5f181a398 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:56:24.398034  6316 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:56:24.403267 15125 raft_consensus.cc:2932] T ac3325a06d8841718158e1f5f181a398 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:56:24.406786  7998 raft_consensus.cc:2932] T ac3325a06d8841718158e1f5f181a398 P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:56:24.410836  5921 catalog_manager.cc:5484] T ac3325a06d8841718158e1f5f181a398 P 94ffd7aaf85b4e17881241b8396fa2f3 reported cstate change: config changed from index 5 to 6, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:56:24.494119  6447 tablet_service.cc:1514] Processing DeleteTablet for tablet ac3325a06d8841718158e1f5f181a398 with delete_type TABLET_DATA_TOMBSTONED (TS 07cdb213c77c4ffda1f999b66b725399 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56564
I20240502 01:56:24.494163  6335 consensus_queue.cc:237] T ac3325a06d8841718158e1f5f181a398 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:56:24.497208  7998 raft_consensus.cc:1261] T ac3325a06d8841718158e1f5f181a398 P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:56:24.498857 15242 consensus_queue.cc:1035] T ac3325a06d8841718158e1f5f181a398 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 01:56:24.500206  6268 consensus_peers.cc:485] T ac3325a06d8841718158e1f5f181a398 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135:40543): Couldn't send request to peer 436822a397664c54b8d1b9eb3bc6cff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ac3325a06d8841718158e1f5f181a398. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:56:24.502913 15242 raft_consensus.cc:2932] T ac3325a06d8841718158e1f5f181a398 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:24.507634  7998 raft_consensus.cc:2932] T ac3325a06d8841718158e1f5f181a398 P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:24.508162  5907 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet ac3325a06d8841718158e1f5f181a398 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 01:56:24.509663  5922 catalog_manager.cc:5484] T ac3325a06d8841718158e1f5f181a398 P 94ffd7aaf85b4e17881241b8396fa2f3 reported cstate change: config changed from index 6 to 7, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:24.591390 15395 tablet_replica.cc:330] T ac3325a06d8841718158e1f5f181a398 P 07cdb213c77c4ffda1f999b66b725399: stopping tablet replica
I20240502 01:56:24.591730 15395 raft_consensus.cc:2229] T ac3325a06d8841718158e1f5f181a398 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:24.592067 15395 raft_consensus.cc:2258] T ac3325a06d8841718158e1f5f181a398 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:24.632943 15395 ts_tablet_manager.cc:1875] T ac3325a06d8841718158e1f5f181a398 P 07cdb213c77c4ffda1f999b66b725399: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:56:24.660425  6447 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:24.669293  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:24.673058 15395 ts_tablet_manager.cc:1888] T ac3325a06d8841718158e1f5f181a398 P 07cdb213c77c4ffda1f999b66b725399: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:56:24.675786 15395 log.cc:1196] T ac3325a06d8841718158e1f5f181a398 P 07cdb213c77c4ffda1f999b66b725399: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal/wals/ac3325a06d8841718158e1f5f181a398
I20240502 01:56:24.690968  5905 catalog_manager.cc:4830] TS 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867): tablet ac3325a06d8841718158e1f5f181a398 (table rebalance_test_table_1 [id=0053d72412224a79be83b7e007b51934]) successfully deleted
I20240502 01:56:24.719751  6316 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:56:24.725476  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:24.795642  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:24.796526  8385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 01:56:24.844641  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:56:24.880138  6136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.134:42517, user_credentials={real_user=slave}} blocked reactor thread for 90704.9us
W20240502 01:56:24.883615  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:24.887953  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 01:56:24.914376  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:56:24.926017  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:56:24.932443  8269 consensus_queue.cc:237] T 89f7cf12d85442749e7d5b259ebca43f P 436822a397664c54b8d1b9eb3bc6cff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:56:24.937775  6066 raft_consensus.cc:1261] T 89f7cf12d85442749e7d5b259ebca43f P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Refusing update from remote peer 436822a397664c54b8d1b9eb3bc6cff4: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:56:24.939257 14380 consensus_queue.cc:1035] T 89f7cf12d85442749e7d5b259ebca43f P 436822a397664c54b8d1b9eb3bc6cff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:56:24.962950 15464 raft_consensus.cc:2932] T 89f7cf12d85442749e7d5b259ebca43f P 436822a397664c54b8d1b9eb3bc6cff4 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:24.963977  6066 raft_consensus.cc:2932] T 89f7cf12d85442749e7d5b259ebca43f P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:24.969867  5920 catalog_manager.cc:5484] T 89f7cf12d85442749e7d5b259ebca43f P 218bc8c1f9ce46059898fc738179adf9 reported cstate change: config changed from index 6 to 7, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New cstate: current_term: 2 leader_uuid: "436822a397664c54b8d1b9eb3bc6cff4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:24.976467  6316 tablet_service.cc:1514] Processing DeleteTablet for tablet 89f7cf12d85442749e7d5b259ebca43f with delete_type TABLET_DATA_TOMBSTONED (TS 94ffd7aaf85b4e17881241b8396fa2f3 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:41134
I20240502 01:56:24.995443 15471 tablet_replica.cc:330] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3: stopping tablet replica
W20240502 01:56:24.996356  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:56:24.996742 15471 raft_consensus.cc:2229] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:24.997815 15471 raft_consensus.cc:2258] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:24.998699 15471 ts_tablet_manager.cc:1875] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:56:25.005012 15471 ts_tablet_manager.cc:1888] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 01:56:25.006981 15471 log.cc:1196] T 89f7cf12d85442749e7d5b259ebca43f P 94ffd7aaf85b4e17881241b8396fa2f3: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/wals/89f7cf12d85442749e7d5b259ebca43f
I20240502 01:56:25.008245  5907 catalog_manager.cc:4830] TS 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245): tablet 89f7cf12d85442749e7d5b259ebca43f (table rebalance_test_table_1 [id=0053d72412224a79be83b7e007b51934]) successfully deleted
I20240502 01:56:25.027592  8269 consensus_queue.cc:237] T 89f7cf12d85442749e7d5b259ebca43f P 436822a397664c54b8d1b9eb3bc6cff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
W20240502 01:56:25.029073  8337 consensus_peers.cc:485] T d8a6db6ae11e4cb8881aee95bdb0e59d P 6d34e50fb5dd4a1db58013d4863b8992 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:56:25.038926  6066 raft_consensus.cc:1261] T 89f7cf12d85442749e7d5b259ebca43f P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Refusing update from remote peer 436822a397664c54b8d1b9eb3bc6cff4: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 01:56:25.046080 14380 consensus_queue.cc:1035] T 89f7cf12d85442749e7d5b259ebca43f P 436822a397664c54b8d1b9eb3bc6cff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 01:56:25.053799  8204 consensus_peers.cc:485] T 89f7cf12d85442749e7d5b259ebca43f P 436822a397664c54b8d1b9eb3bc6cff4 -> Peer 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133:43379): Couldn't send request to peer 60298ffea1dd4c25a38765c2ec9ec9d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 89f7cf12d85442749e7d5b259ebca43f. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:56:25.054447 14380 raft_consensus.cc:2932] T 89f7cf12d85442749e7d5b259ebca43f P 436822a397664c54b8d1b9eb3bc6cff4 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:56:25.058805  5905 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 89f7cf12d85442749e7d5b259ebca43f with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 01:56:25.060861  5922 catalog_manager.cc:5484] T 89f7cf12d85442749e7d5b259ebca43f P 436822a397664c54b8d1b9eb3bc6cff4 reported cstate change: config changed from index 7 to 8, VOTER 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133) added. New cstate: current_term: 2 leader_uuid: "436822a397664c54b8d1b9eb3bc6cff4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
W20240502 01:56:25.068367  8337 consensus_peers.cc:485] T 265a9645e1f141cb88c786b9aef8f90e P 6d34e50fb5dd4a1db58013d4863b8992 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:56:25.070132  6066 raft_consensus.cc:2932] T 89f7cf12d85442749e7d5b259ebca43f P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
W20240502 01:56:25.115170  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:56:25.122361 15501 ts_tablet_manager.cc:902] T ac3325a06d8841718158e1f5f181a398 P 436822a397664c54b8d1b9eb3bc6cff4: Initiating tablet copy from peer 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245)
W20240502 01:56:25.141533  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:56:25.164235 15501 tablet_copy_client.cc:308] T ac3325a06d8841718158e1f5f181a398 P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.131:44245
I20240502 01:56:25.165130  6357 tablet_copy_service.cc:140] P 94ffd7aaf85b4e17881241b8396fa2f3: Received BeginTabletCopySession request for tablet ac3325a06d8841718158e1f5f181a398 from peer 436822a397664c54b8d1b9eb3bc6cff4 ({username='slave'} at 127.25.109.135:37865)
I20240502 01:56:25.165316  6357 tablet_copy_service.cc:161] P 94ffd7aaf85b4e17881241b8396fa2f3: Beginning new tablet copy session on tablet ac3325a06d8841718158e1f5f181a398 from peer 436822a397664c54b8d1b9eb3bc6cff4 at {username='slave'} at 127.25.109.135:37865: session id = 436822a397664c54b8d1b9eb3bc6cff4-ac3325a06d8841718158e1f5f181a398
I20240502 01:56:25.168510  6357 tablet_copy_source_session.cc:205] T ac3325a06d8841718158e1f5f181a398 P 94ffd7aaf85b4e17881241b8396fa2f3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:56:25.169965 15501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac3325a06d8841718158e1f5f181a398. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:56:25.176635 15501 tablet_copy_client.cc:783] T ac3325a06d8841718158e1f5f181a398 P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Starting download of 0 data blocks...
I20240502 01:56:25.176888 15501 tablet_copy_client.cc:647] T ac3325a06d8841718158e1f5f181a398 P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Starting download of 1 WAL segments...
I20240502 01:56:25.178628 15501 tablet_copy_client.cc:523] T ac3325a06d8841718158e1f5f181a398 P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:56:25.182118 15501 tablet_bootstrap.cc:492] T ac3325a06d8841718158e1f5f181a398 P 436822a397664c54b8d1b9eb3bc6cff4: Bootstrap starting.
W20240502 01:56:25.197726  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:56:25.222997  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:56:25.274439  6447 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 01:56:25.295003  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:56:25.296527  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:56:25.300604  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:25.302646 15501 tablet_bootstrap.cc:492] T ac3325a06d8841718158e1f5f181a398 P 436822a397664c54b8d1b9eb3bc6cff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:56:25.303130 15501 tablet_bootstrap.cc:492] T ac3325a06d8841718158e1f5f181a398 P 436822a397664c54b8d1b9eb3bc6cff4: Bootstrap complete.
I20240502 01:56:25.303481 15501 ts_tablet_manager.cc:1367] T ac3325a06d8841718158e1f5f181a398 P 436822a397664c54b8d1b9eb3bc6cff4: Time spent bootstrapping tablet: real 0.121s	user 0.062s	sys 0.028s
I20240502 01:56:25.304322 15501 raft_consensus.cc:348] T ac3325a06d8841718158e1f5f181a398 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:56:25.305357 15501 raft_consensus.cc:729] T ac3325a06d8841718158e1f5f181a398 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 436822a397664c54b8d1b9eb3bc6cff4, State: Initialized, Role: FOLLOWER
I20240502 01:56:25.305637 15501 consensus_queue.cc:260] T ac3325a06d8841718158e1f5f181a398 P 436822a397664c54b8d1b9eb3bc6cff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:56:25.306347 15501 ts_tablet_manager.cc:1398] T ac3325a06d8841718158e1f5f181a398 P 436822a397664c54b8d1b9eb3bc6cff4: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.000s
I20240502 01:56:25.310521  6357 tablet_copy_service.cc:342] P 94ffd7aaf85b4e17881241b8396fa2f3: Request end of tablet copy session 436822a397664c54b8d1b9eb3bc6cff4-ac3325a06d8841718158e1f5f181a398 received from {username='slave'} at 127.25.109.135:37865
I20240502 01:56:25.310662  6357 tablet_copy_service.cc:433] P 94ffd7aaf85b4e17881241b8396fa2f3: ending tablet copy session 436822a397664c54b8d1b9eb3bc6cff4-ac3325a06d8841718158e1f5f181a398 on tablet ac3325a06d8841718158e1f5f181a398 with peer 436822a397664c54b8d1b9eb3bc6cff4
I20240502 01:56:25.328680  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:25.346231  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:56:25.373826  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:56:25.380002  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:25.389359  6316 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:56:25.389497  8385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:25.393570  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 01:56:25.443964  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:56:25.463392 15557 ts_tablet_manager.cc:902] T 89f7cf12d85442749e7d5b259ebca43f P 60298ffea1dd4c25a38765c2ec9ec9d5: Initiating tablet copy from peer 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135:40543)
I20240502 01:56:25.487329 15557 tablet_copy_client.cc:308] T 89f7cf12d85442749e7d5b259ebca43f P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.135:40543
I20240502 01:56:25.499828  8269 raft_consensus.cc:1203] T ac3325a06d8841718158e1f5f181a398 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 01:56:25.575476  8290 tablet_copy_service.cc:140] P 436822a397664c54b8d1b9eb3bc6cff4: Received BeginTabletCopySession request for tablet 89f7cf12d85442749e7d5b259ebca43f from peer 60298ffea1dd4c25a38765c2ec9ec9d5 ({username='slave'} at 127.25.109.133:34271)
I20240502 01:56:25.575722  8290 tablet_copy_service.cc:161] P 436822a397664c54b8d1b9eb3bc6cff4: Beginning new tablet copy session on tablet 89f7cf12d85442749e7d5b259ebca43f from peer 60298ffea1dd4c25a38765c2ec9ec9d5 at {username='slave'} at 127.25.109.133:34271: session id = 60298ffea1dd4c25a38765c2ec9ec9d5-89f7cf12d85442749e7d5b259ebca43f
I20240502 01:56:25.580154  8290 tablet_copy_source_session.cc:205] T 89f7cf12d85442749e7d5b259ebca43f P 436822a397664c54b8d1b9eb3bc6cff4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:56:25.583374 15557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89f7cf12d85442749e7d5b259ebca43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:56:25.592803 15557 tablet_copy_client.cc:783] T 89f7cf12d85442749e7d5b259ebca43f P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Starting download of 0 data blocks...
I20240502 01:56:25.593264 15557 tablet_copy_client.cc:647] T 89f7cf12d85442749e7d5b259ebca43f P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Starting download of 1 WAL segments...
I20240502 01:56:25.596419 15557 tablet_copy_client.cc:523] T 89f7cf12d85442749e7d5b259ebca43f P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:56:25.605383 15557 tablet_bootstrap.cc:492] T 89f7cf12d85442749e7d5b259ebca43f P 60298ffea1dd4c25a38765c2ec9ec9d5: Bootstrap starting.
I20240502 01:56:25.758435 15557 tablet_bootstrap.cc:492] T 89f7cf12d85442749e7d5b259ebca43f P 60298ffea1dd4c25a38765c2ec9ec9d5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:56:25.759068 15557 tablet_bootstrap.cc:492] T 89f7cf12d85442749e7d5b259ebca43f P 60298ffea1dd4c25a38765c2ec9ec9d5: Bootstrap complete.
I20240502 01:56:25.759441 15557 ts_tablet_manager.cc:1367] T 89f7cf12d85442749e7d5b259ebca43f P 60298ffea1dd4c25a38765c2ec9ec9d5: Time spent bootstrapping tablet: real 0.154s	user 0.068s	sys 0.011s
I20240502 01:56:25.760377 15557 raft_consensus.cc:348] T 89f7cf12d85442749e7d5b259ebca43f P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 01:56:25.760675 15557 raft_consensus.cc:729] T 89f7cf12d85442749e7d5b259ebca43f P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60298ffea1dd4c25a38765c2ec9ec9d5, State: Initialized, Role: FOLLOWER
I20240502 01:56:25.761003 15557 consensus_queue.cc:260] T 89f7cf12d85442749e7d5b259ebca43f P 60298ffea1dd4c25a38765c2ec9ec9d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 01:56:25.768879 15557 ts_tablet_manager.cc:1398] T 89f7cf12d85442749e7d5b259ebca43f P 60298ffea1dd4c25a38765c2ec9ec9d5: Time spent starting tablet: real 0.009s	user 0.002s	sys 0.006s
I20240502 01:56:25.770843  8290 tablet_copy_service.cc:342] P 436822a397664c54b8d1b9eb3bc6cff4: Request end of tablet copy session 60298ffea1dd4c25a38765c2ec9ec9d5-89f7cf12d85442749e7d5b259ebca43f received from {username='slave'} at 127.25.109.133:34271
I20240502 01:56:25.771028  8290 tablet_copy_service.cc:433] P 436822a397664c54b8d1b9eb3bc6cff4: ending tablet copy session 60298ffea1dd4c25a38765c2ec9ec9d5-89f7cf12d85442749e7d5b259ebca43f on tablet 89f7cf12d85442749e7d5b259ebca43f with peer 60298ffea1dd4c25a38765c2ec9ec9d5
I20240502 01:56:25.789947  7998 raft_consensus.cc:1203] T 89f7cf12d85442749e7d5b259ebca43f P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8]   Dedup: 2.8->[]
I20240502 01:56:25.862421  8385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:25.869237  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:25.886997  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:25.901895  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:25.904752  6316 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 01:56:25.918300  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:25.969652  6447 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:25.970592  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:26.007727  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:26.021943  6469 consensus_queue.cc:237] T 0039dd405cfc4dcea5e7353169e3cd52 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 01:56:26.028959  8385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:26.030633  7998 raft_consensus.cc:1261] T 0039dd405cfc4dcea5e7353169e3cd52 P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:56:26.031342 14961 consensus_queue.cc:1035] T 0039dd405cfc4dcea5e7353169e3cd52 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:56:26.046185  6447 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:26.050905  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:26.063374  6316 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:56:26.186231  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:26.193742 15701 raft_consensus.cc:2932] T 0039dd405cfc4dcea5e7353169e3cd52 P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:56:26.194698  8001 raft_consensus.cc:2932] T 0039dd405cfc4dcea5e7353169e3cd52 P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:56:26.200356  5916 catalog_manager.cc:5484] T 0039dd405cfc4dcea5e7353169e3cd52 P 60298ffea1dd4c25a38765c2ec9ec9d5 reported cstate change: config changed from index 5 to 6, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New cstate: current_term: 1 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
W20240502 01:56:26.223855  6134 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.135:40543, user_credentials={real_user=slave}} blocked reactor thread for 175356us
I20240502 01:56:26.256455  6469 consensus_queue.cc:237] T 0039dd405cfc4dcea5e7353169e3cd52 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:56:26.257804  6180 tablet_service.cc:1514] Processing DeleteTablet for tablet 0039dd405cfc4dcea5e7353169e3cd52 with delete_type TABLET_DATA_TOMBSTONED (TS a2c90ee440b44b708bccc2e2d5385cc0 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41258
I20240502 01:56:26.259955  7998 raft_consensus.cc:1261] T 0039dd405cfc4dcea5e7353169e3cd52 P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 01:56:26.260588  6400 consensus_peers.cc:485] T 0039dd405cfc4dcea5e7353169e3cd52 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135:40543): Couldn't send request to peer 436822a397664c54b8d1b9eb3bc6cff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0039dd405cfc4dcea5e7353169e3cd52. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:56:26.260644 14961 consensus_queue.cc:1035] T 0039dd405cfc4dcea5e7353169e3cd52 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:56:26.268165 14961 raft_consensus.cc:2932] T 0039dd405cfc4dcea5e7353169e3cd52 P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:26.272449  5905 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 0039dd405cfc4dcea5e7353169e3cd52 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 01:56:26.274449  5915 catalog_manager.cc:5484] T 0039dd405cfc4dcea5e7353169e3cd52 P 07cdb213c77c4ffda1f999b66b725399 reported cstate change: config changed from index 6 to 7, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New cstate: current_term: 1 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:26.274722  7998 raft_consensus.cc:2932] T 0039dd405cfc4dcea5e7353169e3cd52 P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:26.281631 15713 tablet_replica.cc:330] T 0039dd405cfc4dcea5e7353169e3cd52 P a2c90ee440b44b708bccc2e2d5385cc0: stopping tablet replica
I20240502 01:56:26.281869 15713 raft_consensus.cc:2229] T 0039dd405cfc4dcea5e7353169e3cd52 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:26.283428 15713 raft_consensus.cc:2258] T 0039dd405cfc4dcea5e7353169e3cd52 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:26.284263 15713 ts_tablet_manager.cc:1875] T 0039dd405cfc4dcea5e7353169e3cd52 P a2c90ee440b44b708bccc2e2d5385cc0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:56:26.293155 15713 ts_tablet_manager.cc:1888] T 0039dd405cfc4dcea5e7353169e3cd52 P a2c90ee440b44b708bccc2e2d5385cc0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:56:26.293398 15713 log.cc:1196] T 0039dd405cfc4dcea5e7353169e3cd52 P a2c90ee440b44b708bccc2e2d5385cc0: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal/wals/0039dd405cfc4dcea5e7353169e3cd52
I20240502 01:56:26.294397  5908 catalog_manager.cc:4830] TS a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197): tablet 0039dd405cfc4dcea5e7353169e3cd52 (table rebalance_test_table_7 [id=c2701645b8bc497aaf63e970157bdd80]) successfully deleted
I20240502 01:56:26.314426  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:26.395977  6447 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:26.423060  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:26.430639  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:26.443609  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:26.459982  6316 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:56:26.476722  8385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 01:56:26.502710  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:26.558701  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:26.878198 15843 ts_tablet_manager.cc:902] T 0039dd405cfc4dcea5e7353169e3cd52 P 436822a397664c54b8d1b9eb3bc6cff4: Initiating tablet copy from peer 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867)
I20240502 01:56:26.882953  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:26.909740  6447 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:26.949486  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:26.953395 15843 tablet_copy_client.cc:308] T 0039dd405cfc4dcea5e7353169e3cd52 P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.132:36867
I20240502 01:56:26.954255  6489 tablet_copy_service.cc:140] P 07cdb213c77c4ffda1f999b66b725399: Received BeginTabletCopySession request for tablet 0039dd405cfc4dcea5e7353169e3cd52 from peer 436822a397664c54b8d1b9eb3bc6cff4 ({username='slave'} at 127.25.109.135:41247)
I20240502 01:56:26.954452  6489 tablet_copy_service.cc:161] P 07cdb213c77c4ffda1f999b66b725399: Beginning new tablet copy session on tablet 0039dd405cfc4dcea5e7353169e3cd52 from peer 436822a397664c54b8d1b9eb3bc6cff4 at {username='slave'} at 127.25.109.135:41247: session id = 436822a397664c54b8d1b9eb3bc6cff4-0039dd405cfc4dcea5e7353169e3cd52
I20240502 01:56:26.958969  6489 tablet_copy_source_session.cc:205] T 0039dd405cfc4dcea5e7353169e3cd52 P 07cdb213c77c4ffda1f999b66b725399: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:56:26.960799 15843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0039dd405cfc4dcea5e7353169e3cd52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:56:26.972815 15843 tablet_copy_client.cc:783] T 0039dd405cfc4dcea5e7353169e3cd52 P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Starting download of 0 data blocks...
I20240502 01:56:26.973176 15843 tablet_copy_client.cc:647] T 0039dd405cfc4dcea5e7353169e3cd52 P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Starting download of 1 WAL segments...
I20240502 01:56:26.975854 15843 tablet_copy_client.cc:523] T 0039dd405cfc4dcea5e7353169e3cd52 P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:56:26.994146  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:56:27.003645  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:27.009284 15843 tablet_bootstrap.cc:492] T 0039dd405cfc4dcea5e7353169e3cd52 P 436822a397664c54b8d1b9eb3bc6cff4: Bootstrap starting.
I20240502 01:56:27.051788  8385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:27.051774  6316 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:56:27.054509  6469 consensus_queue.cc:237] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 01:56:27.057288  8001 raft_consensus.cc:1261] T 02b106ec57b94e44a29947a80a030a23 P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:56:27.058171 14961 consensus_queue.cc:1035] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 01:56:27.078346  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:56:27.085496  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:27.113343 15702 raft_consensus.cc:2932] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:56:27.117105  7998 raft_consensus.cc:2932] T 02b106ec57b94e44a29947a80a030a23 P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:56:27.122167  5915 catalog_manager.cc:5484] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 reported cstate change: config changed from index 5 to 6, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New cstate: current_term: 1 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:56:27.155939 15843 tablet_bootstrap.cc:492] T 0039dd405cfc4dcea5e7353169e3cd52 P 436822a397664c54b8d1b9eb3bc6cff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:56:27.156512 15843 tablet_bootstrap.cc:492] T 0039dd405cfc4dcea5e7353169e3cd52 P 436822a397664c54b8d1b9eb3bc6cff4: Bootstrap complete.
I20240502 01:56:27.156905 15843 ts_tablet_manager.cc:1367] T 0039dd405cfc4dcea5e7353169e3cd52 P 436822a397664c54b8d1b9eb3bc6cff4: Time spent bootstrapping tablet: real 0.157s	user 0.053s	sys 0.000s
I20240502 01:56:27.158314 15843 raft_consensus.cc:348] T 0039dd405cfc4dcea5e7353169e3cd52 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:56:27.158618 15843 raft_consensus.cc:729] T 0039dd405cfc4dcea5e7353169e3cd52 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 436822a397664c54b8d1b9eb3bc6cff4, State: Initialized, Role: FOLLOWER
I20240502 01:56:27.158910 15843 consensus_queue.cc:260] T 0039dd405cfc4dcea5e7353169e3cd52 P 436822a397664c54b8d1b9eb3bc6cff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:56:27.159516 15843 ts_tablet_manager.cc:1398] T 0039dd405cfc4dcea5e7353169e3cd52 P 436822a397664c54b8d1b9eb3bc6cff4: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 01:56:27.160948  6489 tablet_copy_service.cc:342] P 07cdb213c77c4ffda1f999b66b725399: Request end of tablet copy session 436822a397664c54b8d1b9eb3bc6cff4-0039dd405cfc4dcea5e7353169e3cd52 received from {username='slave'} at 127.25.109.135:41247
I20240502 01:56:27.161074  6489 tablet_copy_service.cc:433] P 07cdb213c77c4ffda1f999b66b725399: ending tablet copy session 436822a397664c54b8d1b9eb3bc6cff4-0039dd405cfc4dcea5e7353169e3cd52 on tablet 0039dd405cfc4dcea5e7353169e3cd52 with peer 436822a397664c54b8d1b9eb3bc6cff4
I20240502 01:56:27.211635  6180 tablet_service.cc:1514] Processing DeleteTablet for tablet 02b106ec57b94e44a29947a80a030a23 with delete_type TABLET_DATA_TOMBSTONED (TS a2c90ee440b44b708bccc2e2d5385cc0 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41258
I20240502 01:56:27.212893  6469 consensus_queue.cc:237] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:56:27.217482  7998 raft_consensus.cc:1261] T 02b106ec57b94e44a29947a80a030a23 P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:56:27.218211 15702 consensus_queue.cc:1035] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 01:56:27.219321  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 02b106ec57b94e44a29947a80a030a23. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:56:27.223219 15702 raft_consensus.cc:2932] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:27.224601  7998 raft_consensus.cc:2932] T 02b106ec57b94e44a29947a80a030a23 P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:27.249367  5905 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 02b106ec57b94e44a29947a80a030a23 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 01:56:27.254202  5915 catalog_manager.cc:5484] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 reported cstate change: config changed from index 6 to 7, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New cstate: current_term: 1 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:27.271955 15920 tablet_replica.cc:330] T 02b106ec57b94e44a29947a80a030a23 P a2c90ee440b44b708bccc2e2d5385cc0: stopping tablet replica
I20240502 01:56:27.286753 15920 raft_consensus.cc:2229] T 02b106ec57b94e44a29947a80a030a23 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:27.288322 15920 raft_consensus.cc:2258] T 02b106ec57b94e44a29947a80a030a23 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:27.293964 15920 ts_tablet_manager.cc:1875] T 02b106ec57b94e44a29947a80a030a23 P a2c90ee440b44b708bccc2e2d5385cc0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:56:27.304726 15920 ts_tablet_manager.cc:1888] T 02b106ec57b94e44a29947a80a030a23 P a2c90ee440b44b708bccc2e2d5385cc0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:56:27.305251 15920 log.cc:1196] T 02b106ec57b94e44a29947a80a030a23 P a2c90ee440b44b708bccc2e2d5385cc0: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal/wals/02b106ec57b94e44a29947a80a030a23
I20240502 01:56:27.306281  5908 catalog_manager.cc:4830] TS a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197): tablet 02b106ec57b94e44a29947a80a030a23 (table rebalance_test_table_2 [id=ef7ebedfbbff4054b074bf7a7622b632]) successfully deleted
I20240502 01:56:27.357784  8269 raft_consensus.cc:1203] T 0039dd405cfc4dcea5e7353169e3cd52 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
W20240502 01:56:27.412796  8337 consensus_peers.cc:485] T d8a6db6ae11e4cb8881aee95bdb0e59d P 6d34e50fb5dd4a1db58013d4863b8992 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:56:27.444686  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:56:27.504344  6335 consensus_queue.cc:237] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:56:27.513566  8404 raft_consensus.cc:1261] T 9a5f486852554e2fa2d07f7b9d686ced P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:56:27.514685 15689 consensus_queue.cc:1035] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 01:56:27.517962  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:27.549075  8404 raft_consensus.cc:2932] T 9a5f486852554e2fa2d07f7b9d686ced P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:27.555565  5916 catalog_manager.cc:5484] T 9a5f486852554e2fa2d07f7b9d686ced P 6d34e50fb5dd4a1db58013d4863b8992 reported cstate change: config changed from index 5 to 6, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:27.561949  6180 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a5f486852554e2fa2d07f7b9d686ced with delete_type TABLET_DATA_TOMBSTONED (TS a2c90ee440b44b708bccc2e2d5385cc0 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41258
I20240502 01:56:27.562386 15920 tablet_replica.cc:330] T 9a5f486852554e2fa2d07f7b9d686ced P a2c90ee440b44b708bccc2e2d5385cc0: stopping tablet replica
I20240502 01:56:27.562595 15920 raft_consensus.cc:2229] T 9a5f486852554e2fa2d07f7b9d686ced P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:27.562942 15920 raft_consensus.cc:2258] T 9a5f486852554e2fa2d07f7b9d686ced P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:27.547839 16014 raft_consensus.cc:2932] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:27.563792 15920 ts_tablet_manager.cc:1875] T 9a5f486852554e2fa2d07f7b9d686ced P a2c90ee440b44b708bccc2e2d5385cc0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:56:27.571453 15920 ts_tablet_manager.cc:1888] T 9a5f486852554e2fa2d07f7b9d686ced P a2c90ee440b44b708bccc2e2d5385cc0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:56:27.571697 15920 log.cc:1196] T 9a5f486852554e2fa2d07f7b9d686ced P a2c90ee440b44b708bccc2e2d5385cc0: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal/wals/9a5f486852554e2fa2d07f7b9d686ced
I20240502 01:56:27.572628  5908 catalog_manager.cc:4830] TS a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197): tablet 9a5f486852554e2fa2d07f7b9d686ced (table rebalance_test_table_0 [id=ef6f7a2e933049e5982deec12e9a670a]) successfully deleted
I20240502 01:56:27.600271  6335 consensus_queue.cc:237] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:56:27.603431  8404 raft_consensus.cc:1261] T 9a5f486852554e2fa2d07f7b9d686ced P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:56:27.604030 15689 consensus_queue.cc:1035] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:56:27.606979  8406 raft_consensus.cc:2932] T 9a5f486852554e2fa2d07f7b9d686ced P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
W20240502 01:56:27.611827  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:56:27.613466  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:56:27.617506  5916 catalog_manager.cc:5484] T 9a5f486852554e2fa2d07f7b9d686ced P 6d34e50fb5dd4a1db58013d4863b8992 reported cstate change: config changed from index 6 to 7, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
W20240502 01:56:27.621315  6268 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135:40543): Couldn't send request to peer 436822a397664c54b8d1b9eb3bc6cff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9a5f486852554e2fa2d07f7b9d686ced. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:56:27.622330 16014 raft_consensus.cc:2932] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
W20240502 01:56:27.633648  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:56:27.634395  5907 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 9a5f486852554e2fa2d07f7b9d686ced with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
W20240502 01:56:27.641875  8337 consensus_peers.cc:485] T 265a9645e1f141cb88c786b9aef8f90e P 6d34e50fb5dd4a1db58013d4863b8992 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:56:27.657244  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:56:27.726562 16066 ts_tablet_manager.cc:902] T 02b106ec57b94e44a29947a80a030a23 P 6d34e50fb5dd4a1db58013d4863b8992: Initiating tablet copy from peer 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867)
I20240502 01:56:27.748538 16066 tablet_copy_client.cc:308] T 02b106ec57b94e44a29947a80a030a23 P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.132:36867
I20240502 01:56:27.749418  6489 tablet_copy_service.cc:140] P 07cdb213c77c4ffda1f999b66b725399: Received BeginTabletCopySession request for tablet 02b106ec57b94e44a29947a80a030a23 from peer 6d34e50fb5dd4a1db58013d4863b8992 ({username='slave'} at 127.25.109.136:34855)
I20240502 01:56:27.749603  6489 tablet_copy_service.cc:161] P 07cdb213c77c4ffda1f999b66b725399: Beginning new tablet copy session on tablet 02b106ec57b94e44a29947a80a030a23 from peer 6d34e50fb5dd4a1db58013d4863b8992 at {username='slave'} at 127.25.109.136:34855: session id = 6d34e50fb5dd4a1db58013d4863b8992-02b106ec57b94e44a29947a80a030a23
I20240502 01:56:27.752501  6489 tablet_copy_source_session.cc:205] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:56:27.756294 16066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02b106ec57b94e44a29947a80a030a23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:56:27.772176 16066 tablet_copy_client.cc:783] T 02b106ec57b94e44a29947a80a030a23 P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Starting download of 0 data blocks...
I20240502 01:56:27.772672 16066 tablet_copy_client.cc:647] T 02b106ec57b94e44a29947a80a030a23 P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Starting download of 1 WAL segments...
I20240502 01:56:27.782047 16066 tablet_copy_client.cc:523] T 02b106ec57b94e44a29947a80a030a23 P 6d34e50fb5dd4a1db58013d4863b8992: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240502 01:56:27.785249  6000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240502 01:56:27.786248  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:56:27.787021  6136 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 01:56:27.788251  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:56:27.791512 16066 tablet_bootstrap.cc:492] T 02b106ec57b94e44a29947a80a030a23 P 6d34e50fb5dd4a1db58013d4863b8992: Bootstrap starting.
W20240502 01:56:27.798877  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:56:27.826925  6402 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240502 01:56:27.828178  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:56:27.842186  6447 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:27.859675  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:56:27.880816  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:56:27.900687  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:27.956843  8385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 01:56:27.979807  8337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240502 01:56:27.980670  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:27.985329  6316 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 01:56:28.010156  6402 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.131:44245, user_credentials={real_user=slave}} blocked reactor thread for 43569.3us
W20240502 01:56:28.045156  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:28.047658  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 01:56:28.068732  6270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 01:56:28.071012  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:56:28.079118  6447 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:28.150081  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:28.160862  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:28.194869  8385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:28.212076  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:28.267948  6316 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:56:28.271577 16066 tablet_bootstrap.cc:492] T 02b106ec57b94e44a29947a80a030a23 P 6d34e50fb5dd4a1db58013d4863b8992: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:56:28.274263 16066 tablet_bootstrap.cc:492] T 02b106ec57b94e44a29947a80a030a23 P 6d34e50fb5dd4a1db58013d4863b8992: Bootstrap complete.
I20240502 01:56:28.274606 16066 ts_tablet_manager.cc:1367] T 02b106ec57b94e44a29947a80a030a23 P 6d34e50fb5dd4a1db58013d4863b8992: Time spent bootstrapping tablet: real 0.483s	user 0.055s	sys 0.004s
I20240502 01:56:28.275390 16066 raft_consensus.cc:348] T 02b106ec57b94e44a29947a80a030a23 P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:56:28.275656 16066 raft_consensus.cc:729] T 02b106ec57b94e44a29947a80a030a23 P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d34e50fb5dd4a1db58013d4863b8992, State: Initialized, Role: FOLLOWER
I20240502 01:56:28.276047 16066 consensus_queue.cc:260] T 02b106ec57b94e44a29947a80a030a23 P 6d34e50fb5dd4a1db58013d4863b8992 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:56:28.279577 16149 ts_tablet_manager.cc:902] T 9a5f486852554e2fa2d07f7b9d686ced P 436822a397664c54b8d1b9eb3bc6cff4: Initiating tablet copy from peer 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245)
I20240502 01:56:28.289994 16066 ts_tablet_manager.cc:1398] T 02b106ec57b94e44a29947a80a030a23 P 6d34e50fb5dd4a1db58013d4863b8992: Time spent starting tablet: real 0.015s	user 0.002s	sys 0.000s
I20240502 01:56:28.291121  6489 tablet_copy_service.cc:342] P 07cdb213c77c4ffda1f999b66b725399: Request end of tablet copy session 6d34e50fb5dd4a1db58013d4863b8992-02b106ec57b94e44a29947a80a030a23 received from {username='slave'} at 127.25.109.136:34855
I20240502 01:56:28.291258  6489 tablet_copy_service.cc:433] P 07cdb213c77c4ffda1f999b66b725399: ending tablet copy session 6d34e50fb5dd4a1db58013d4863b8992-02b106ec57b94e44a29947a80a030a23 on tablet 02b106ec57b94e44a29947a80a030a23 with peer 6d34e50fb5dd4a1db58013d4863b8992
I20240502 01:56:28.318805  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:28.378084 16149 tablet_copy_client.cc:308] T 9a5f486852554e2fa2d07f7b9d686ced P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.131:44245
I20240502 01:56:28.379431  6357 tablet_copy_service.cc:140] P 94ffd7aaf85b4e17881241b8396fa2f3: Received BeginTabletCopySession request for tablet 9a5f486852554e2fa2d07f7b9d686ced from peer 436822a397664c54b8d1b9eb3bc6cff4 ({username='slave'} at 127.25.109.135:37865)
I20240502 01:56:28.379602  6357 tablet_copy_service.cc:161] P 94ffd7aaf85b4e17881241b8396fa2f3: Beginning new tablet copy session on tablet 9a5f486852554e2fa2d07f7b9d686ced from peer 436822a397664c54b8d1b9eb3bc6cff4 at {username='slave'} at 127.25.109.135:37865: session id = 436822a397664c54b8d1b9eb3bc6cff4-9a5f486852554e2fa2d07f7b9d686ced
I20240502 01:56:28.382582  6357 tablet_copy_source_session.cc:205] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:56:28.385387 16149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a5f486852554e2fa2d07f7b9d686ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:56:28.403831  6447 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:28.409554 16149 tablet_copy_client.cc:783] T 9a5f486852554e2fa2d07f7b9d686ced P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Starting download of 0 data blocks...
I20240502 01:56:28.409924 16149 tablet_copy_client.cc:647] T 9a5f486852554e2fa2d07f7b9d686ced P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Starting download of 1 WAL segments...
I20240502 01:56:28.411842 16149 tablet_copy_client.cc:523] T 9a5f486852554e2fa2d07f7b9d686ced P 436822a397664c54b8d1b9eb3bc6cff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:56:28.415915 16149 tablet_bootstrap.cc:492] T 9a5f486852554e2fa2d07f7b9d686ced P 436822a397664c54b8d1b9eb3bc6cff4: Bootstrap starting.
I20240502 01:56:28.443790  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:28.474575 16149 tablet_bootstrap.cc:492] T 9a5f486852554e2fa2d07f7b9d686ced P 436822a397664c54b8d1b9eb3bc6cff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:56:28.475085 16149 tablet_bootstrap.cc:492] T 9a5f486852554e2fa2d07f7b9d686ced P 436822a397664c54b8d1b9eb3bc6cff4: Bootstrap complete.
I20240502 01:56:28.475386 16149 ts_tablet_manager.cc:1367] T 9a5f486852554e2fa2d07f7b9d686ced P 436822a397664c54b8d1b9eb3bc6cff4: Time spent bootstrapping tablet: real 0.060s	user 0.039s	sys 0.013s
I20240502 01:56:28.476162 16149 raft_consensus.cc:348] T 9a5f486852554e2fa2d07f7b9d686ced P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:56:28.476462 16149 raft_consensus.cc:729] T 9a5f486852554e2fa2d07f7b9d686ced P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 436822a397664c54b8d1b9eb3bc6cff4, State: Initialized, Role: FOLLOWER
I20240502 01:56:28.476758 16149 consensus_queue.cc:260] T 9a5f486852554e2fa2d07f7b9d686ced P 436822a397664c54b8d1b9eb3bc6cff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:56:28.477334 16149 ts_tablet_manager.cc:1398] T 9a5f486852554e2fa2d07f7b9d686ced P 436822a397664c54b8d1b9eb3bc6cff4: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 01:56:28.478300  6357 tablet_copy_service.cc:342] P 94ffd7aaf85b4e17881241b8396fa2f3: Request end of tablet copy session 436822a397664c54b8d1b9eb3bc6cff4-9a5f486852554e2fa2d07f7b9d686ced received from {username='slave'} at 127.25.109.135:37865
I20240502 01:56:28.478415  6357 tablet_copy_service.cc:433] P 94ffd7aaf85b4e17881241b8396fa2f3: ending tablet copy session 436822a397664c54b8d1b9eb3bc6cff4-9a5f486852554e2fa2d07f7b9d686ced on tablet 9a5f486852554e2fa2d07f7b9d686ced with peer 436822a397664c54b8d1b9eb3bc6cff4
I20240502 01:56:28.495613  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:28.559370  8385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:28.572466  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:28.608793  6316 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 01:56:28.622642  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:28.638550  8406 raft_consensus.cc:1203] T 02b106ec57b94e44a29947a80a030a23 P 6d34e50fb5dd4a1db58013d4863b8992 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 01:56:28.648463  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:28.672291  6447 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:28.672802  8269 raft_consensus.cc:1203] T 9a5f486852554e2fa2d07f7b9d686ced P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 01:56:28.691941  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:28.729590  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:28.749727  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:28.751961  6316 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:56:28.757108  8385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:28.816646  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 01:56:28.867745  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:28.983283  6447 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:28.984055  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:29.001089  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:29.029234  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:29.106823  8385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:29.157840  6316 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:56:29.212316  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:29.330665  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:29.348100  6447 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:29.360411  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:29.382436  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:29.385221  8385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:29.400911  6316 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:56:29.438702  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 01:56:29.504573  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:56:29.586171  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:56:29.823681  8337 consensus_peers.cc:485] T d8a6db6ae11e4cb8881aee95bdb0e59d P 6d34e50fb5dd4a1db58013d4863b8992 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:56:29.940847  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:56:29.991160  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:29.999780  6447 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 01:56:30.019776  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:56:30.045532  8457 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240502 01:55:43.505016  8460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:43.505378  8460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:43.522821  8460 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:55:46.463840  8460 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 01:55:48.432905  8460 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c02dbeef0b7241e78a12e4bd7ef79e47: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:55:48.433363  8460 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0053d72412224a79be83b7e007b51934: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 436822a397664c54b8d1b9eb3bc6cff4
W20240502 01:55:48.433549  8460 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f406d4a60a5e40c485796d37ff4d76bc: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 60298ffea1dd4c25a38765c2ec9ec9d5
W20240502 01:55:48.433697  8460 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1ed4a88787d6401481245c552170a456: could not find any suitable replica to move from server 218bc8c1f9ce46059898fc738179adf9 to server 436822a397664c54b8d1b9eb3bc6cff4
W20240502 01:55:48.433905  8460 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5606c3da105b4b86868c6c51ecdd39aa: could not find any suitable replica to move from server 218bc8c1f9ce46059898fc738179adf9 to server 60298ffea1dd4c25a38765c2ec9ec9d5
W20240502 01:55:48.433995  8460 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c02dbeef0b7241e78a12e4bd7ef79e47: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 436822a397664c54b8d1b9eb3bc6cff4
W20240502 01:55:48.434234  8460 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ef7ebedfbbff4054b074bf7a7622b632: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:55:48.434373  8460 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ef6f7a2e933049e5982deec12e9a670a: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:55:48.434454  8460 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1ed4a88787d6401481245c552170a456: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 60298ffea1dd4c25a38765c2ec9ec9d5
W20240502 01:55:48.434541  8460 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 63a163be603345428cc84c86b96467f0: could not find any suitable replica to move from server 218bc8c1f9ce46059898fc738179adf9 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:55:48.434623  8460 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fcde569cfdf747c2aaecac7d265b4b1f: could not find any suitable replica to move from server 218bc8c1f9ce46059898fc738179adf9 to server 60298ffea1dd4c25a38765c2ec9ec9d5
W20240502 01:55:48.434713  8460 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c2701645b8bc497aaf63e970157bdd80: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 436822a397664c54b8d1b9eb3bc6cff4
W20240502 01:55:48.434806  8460 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0053d72412224a79be83b7e007b51934: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 60298ffea1dd4c25a38765c2ec9ec9d5
W20240502 01:55:48.434944  8460 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5606c3da105b4b86868c6c51ecdd39aa: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:55:48.435026  8460 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ef7ebedfbbff4054b074bf7a7622b632: could not find any suitable replica to move from server 218bc8c1f9ce46059898fc738179adf9 to server 436822a397664c54b8d1b9eb3bc6cff4
I20240502 01:55:48.958156  8460 rebalancer_tool.cc:1084] tablet 0cecd1b14e474dee825fae040ce1b31e: '218bc8c1f9ce46059898fc738179adf9' -> '9db17f84d37944098cbeb9e0d8456efb' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240502 01:55:48.958477  8460 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:55:50.967268  8460 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0053d72412224a79be83b7e007b51934: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:55:50.967713  8460 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5606c3da105b4b86868c6c51ecdd39aa: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 436822a397664c54b8d1b9eb3bc6cff4
W20240502 01:55:50.967993  8460 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c02dbeef0b7241e78a12e4bd7ef79e47: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:55:50.968111  8460 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f406d4a60a5e40c485796d37ff4d76bc: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:55:50.968242  8460 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1ed4a88787d6401481245c552170a456: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 436822a397664c54b8d1b9eb3bc6cff4
W20240502 01:55:50.968328  8460 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 63a163be603345428cc84c86b96467f0: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:55:50.968420  8460 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fcde569cfdf747c2aaecac7d265b4b1f: could not find any suitable replica to move from server 218bc8c1f9ce46059898fc738179adf9 to server 60298ffea1dd4c25a38765c2ec9ec9d5
W20240502 01:55:50.968571  8460 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ef7ebedfbbff4054b074bf7a7622b632: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:55:50.968763  8460 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 63a163be603345428cc84c86b96467f0: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:55:50.968910  8460 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c2701645b8bc497aaf63e970157bdd80: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:55:50.969017  8460 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ef6f7a2e933049e5982deec12e9a670a: could not find any suitable replica to move from server 218bc8c1f9ce46059898fc738179adf9 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:55:50.969095  8460 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0053d72412224a79be83b7e007b51934: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 60298ffea1dd4c25a38765c2ec9ec9d5
W20240502 01:55:50.969185  8460 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5606c3da105b4b86868c6c51ecdd39aa: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 436822a397664c54b8d1b9eb3bc6cff4
W20240502 01:55:50.969257  8460 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c02dbeef0b7241e78a12e4bd7ef79e47: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:55:50.969352  8460 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ef7ebedfbbff4054b074bf7a7622b632: could not find any suitable replica to move from server 218bc8c1f9ce46059898fc738179adf9 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:55:50.969432  8460 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f406d4a60a5e40c485796d37ff4d76bc: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 436822a397664c54b8d1b9eb3bc6cff4
W20240502 01:55:50.969520  8460 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1ed4a88787d6401481245c552170a456: could not find any suitable replica to move from server 218bc8c1f9ce46059898fc738179adf9 to server 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:55:50.969594  8460 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fcde569cfdf747c2aaecac7d265b4b1f: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 60298ffea1dd4c25a38765c2ec9ec9d5
I20240502 01:55:51.548249  8460 rebalancer_tool.cc:1074] tablet 73ed91df7ff04846991c95b30a86ea7e: '218bc8c1f9ce46059898fc738179adf9' -> '6d34e50fb5dd4a1db58013d4863b8992' move scheduled
I20240502 01:55:51.977378  8460 rebalancer_tool.cc:1074] tablet 4eb91f5df851459b8ecf09cbe34fefa5: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '9db17f84d37944098cbeb9e0d8456efb' move scheduled
I20240502 01:55:52.745482  8460 rebalancer_tool.cc:1074] tablet 086509f1ee0a4a81b2f8194b50dbe159: '218bc8c1f9ce46059898fc738179adf9' -> '436822a397664c54b8d1b9eb3bc6cff4' move scheduled
I20240502 01:55:53.363037  8460 rebalancer_tool.cc:1074] tablet 6984d638991442e3adaed1b24bee95e9: '94ffd7aaf85b4e17881241b8396fa2f3' -> '60298ffea1dd4c25a38765c2ec9ec9d5' move scheduled
I20240502 01:55:53.717557  8460 rebalancer_tool.cc:1074] tablet c58d6b2401c047cd9af56a66c2e00105: '07cdb213c77c4ffda1f999b66b725399' -> '6d34e50fb5dd4a1db58013d4863b8992' move scheduled
I20240502 01:55:54.205119  8460 rebalancer_tool.cc:1074] tablet 5be45a68cff24a75a03b2fcd4b7851f9: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '436822a397664c54b8d1b9eb3bc6cff4' move scheduled
I20240502 01:55:54.585824  8460 rebalancer_tool.cc:1074] tablet afa5561f8c0e4c0fa9bbbd5da61348c7: '07cdb213c77c4ffda1f999b66b725399' -> '9db17f84d37944098cbeb9e0d8456efb' move scheduled
I20240502 01:55:55.060757  8460 rebalancer_tool.cc:1074] tablet 0039dd405cfc4dcea5e7353169e3cd52: '94ffd7aaf85b4e17881241b8396fa2f3' -> '60298ffea1dd4c25a38765c2ec9ec9d5' move scheduled
I20240502 01:55:55.870826  8460 rebalancer_tool.cc:1074] tablet 3a7698f6bd4349ae832a22c853d76fea: '218bc8c1f9ce46059898fc738179adf9' -> '6d34e50fb5dd4a1db58013d4863b8992' move scheduled
I20240502 01:55:56.386344  8460 rebalancer_tool.cc:1074] tablet 866f87ea9f7f48edb667b1b2eb3fa239: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '436822a397664c54b8d1b9eb3bc6cff4' move scheduled
I20240502 01:55:56.615113  8460 rebalancer_tool.cc:1084] tablet 899963a8260a42c4b73fb80eda2ecaa9: '07cdb213c77c4ffda1f999b66b725399' -> '9db17f84d37944098cbeb9e0d8456efb' move ignored: Illegal state: tablet 899963a8260a42c4b73fb80eda2ecaa9: replica 9db17f84d37944098cbeb9e0d8456efb is already present
I20240502 01:55:56.615311  8460 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:55:58.143146  8460 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0053d72412224a79be83b7e007b51934: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 60298ffea1dd4c25a38765c2ec9ec9d5
W20240502 01:55:58.143465  8460 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c2701645b8bc497aaf63e970157bdd80: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 436822a397664c54b8d1b9eb3bc6cff4
W20240502 01:55:58.143575  8460 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c2701645b8bc497aaf63e970157bdd80: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:55:58.143736  8460 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 63a163be603345428cc84c86b96467f0: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:55:58.143894  8460 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f406d4a60a5e40c485796d37ff4d76bc: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:55:58.144034  8460 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1ed4a88787d6401481245c552170a456: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 436822a397664c54b8d1b9eb3bc6cff4
W20240502 01:55:58.144199  8460 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fcde569cfdf747c2aaecac7d265b4b1f: could not find any suitable replica to move from server 218bc8c1f9ce46059898fc738179adf9 to server 436822a397664c54b8d1b9eb3bc6cff4
W20240502 01:55:58.144296  8460 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c02dbeef0b7241e78a12e4bd7ef79e47: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:55:58.144452  8460 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5606c3da105b4b86868c6c51ecdd39aa: could not find any suitable replica to move from server 218bc8c1f9ce46059898fc738179adf9 to server 60298ffea1dd4c25a38765c2ec9ec9d5
W20240502 01:55:58.144551  8460 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ef6f7a2e933049e5982deec12e9a670a: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:55:58.315613  8460 rebalancer_tool.cc:1074] tablet f309b40931dd444091869119eab33bbc: '94ffd7aaf85b4e17881241b8396fa2f3' -> '' move scheduled
I20240502 01:55:58.386324  8460 rebalancer_tool.cc:1074] tablet 99e74a49b42745888058db67049dc682: '07cdb213c77c4ffda1f999b66b725399' -> '' move scheduled
I20240502 01:55:58.878783  8460 rebalancer_tool.cc:1074] tablet 4f4088f3523f4cd0afdf0de0f447c7af: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '9db17f84d37944098cbeb9e0d8456efb' move scheduled
I20240502 01:55:58.929821  8460 rebalancer_tool.cc:1074] tablet 74eadf4c939c4c6383732cd74c5ec9d6: '218bc8c1f9ce46059898fc738179adf9' -> '' move scheduled
I20240502 01:55:59.093045  8460 rebalancer_tool.cc:1074] tablet e4d4a1bd80f44b668a2d504e47fb744a: '94ffd7aaf85b4e17881241b8396fa2f3' -> '' move scheduled
I20240502 01:55:59.260776  8460 rebalancer_tool.cc:1074] tablet 899963a8260a42c4b73fb80eda2ecaa9: '07cdb213c77c4ffda1f999b66b725399' -> '' move scheduled
W20240502 01:55:59.406217 11195 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.25.109.130:38197, user_credentials={real_user=slave}} blocked reactor thread for 54166.7us
I20240502 01:56:06.582463  8460 rebalancer_tool.cc:1136] tablet 3a7698f6bd4349ae832a22c853d76fea: '218bc8c1f9ce46059898fc738179adf9' -> '6d34e50fb5dd4a1db58013d4863b8992' move completed: OK
I20240502 01:56:13.240757  8460 rebalancer_tool.cc:1136] tablet afa5561f8c0e4c0fa9bbbd5da61348c7: '07cdb213c77c4ffda1f999b66b725399' -> '9db17f84d37944098cbeb9e0d8456efb' move completed: OK
I20240502 01:56:17.778406  8460 rebalancer_tool.cc:1136] tablet 899963a8260a42c4b73fb80eda2ecaa9: '07cdb213c77c4ffda1f999b66b725399' -> '' move completed: OK
I20240502 01:56:19.935222  8460 rebalancer_tool.cc:1136] tablet c58d6b2401c047cd9af56a66c2e00105: '07cdb213c77c4ffda1f999b66b725399' -> '6d34e50fb5dd4a1db58013d4863b8992' move completed: OK
I20240502 01:56:20.201074  8460 rebalancer_tool.cc:1136] tablet 5be45a68cff24a75a03b2fcd4b7851f9: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '436822a397664c54b8d1b9eb3bc6cff4' move completed: OK
I20240502 01:56:22.645486  8460 rebalancer_tool.cc:1136] tablet 6984d638991442e3adaed1b24bee95e9: '94ffd7aaf85b4e17881241b8396fa2f3' -> '60298ffea1dd4c25a38765c2ec9ec9d5' move completed: OK
I20240502 01:56:22.727994  8460 rebalancer_tool.cc:1074] tablet ac3325a06d8841718158e1f5f181a398: '07cdb213c77c4ffda1f999b66b725399' -> '' move scheduled
I20240502 01:56:24.640723  8460 rebalancer_tool.cc:1136] tablet e4d4a1bd80f44b668a2d504e47fb744a: '94ffd7aaf85b4e17881241b8396fa2f3' -> '' move completed: OK
I20240502 01:56:24.841924  8460 rebalancer_tool.cc:1136] tablet 74eadf4c939c4c6383732cd74c5ec9d6: '218bc8c1f9ce46059898fc738179adf9' -> '' move completed: OK
I20240502 01:56:24.866515  8460 rebalancer_tool.cc:1136] tablet 866f87ea9f7f48edb667b1b2eb3fa239: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '436822a397664c54b8d1b9eb3bc6cff4' move completed: OK
I20240502 01:56:24.947853  8460 rebalancer_tool.cc:1136] tablet 4f4088f3523f4cd0afdf0de0f447c7af: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '9db17f84d37944098cbeb9e0d8456efb' move completed: OK
I20240502 01:56:25.010496  8460 rebalancer_tool.cc:1136] tablet 99e74a49b42745888058db67049dc682: '07cdb213c77c4ffda1f999b66b725399' -> '' move completed: OK
I20240502 01:56:25.200021  8460 rebalancer_tool.cc:1136] tablet 0039dd405cfc4dcea5e7353169e3cd52: '94ffd7aaf85b4e17881241b8396fa2f3' -> '60298ffea1dd4c25a38765c2ec9ec9d5' move completed: OK
I20240502 01:56:25.289295  8460 rebalancer_tool.cc:1136] tablet 4eb91f5df851459b8ecf09cbe34fefa5: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '9db17f84d37944098cbeb9e0d8456efb' move completed: OK
I20240502 01:56:25.349800  8460 rebalancer_tool.cc:1136] tablet 73ed91df7ff04846991c95b30a86ea7e: '218bc8c1f9ce46059898fc738179adf9' -> '6d34e50fb5dd4a1db58013d4863b8992' move completed: OK
I20240502 01:56:25.507023  8460 rebalancer_tool.cc:1136] tablet 086509f1ee0a4a81b2f8194b50dbe159: '218bc8c1f9ce46059898fc738179adf9' -> '436822a397664c54b8d1b9eb3bc6cff4' move completed: OK
I20240502 01:56:25.614384  8460 rebalancer_tool.cc:1136] tablet f309b40931dd444091869119eab33bbc: '94ffd7aaf85b4e17881241b8396fa2f3' -> '' move completed: OK
I20240502 01:56:25.707198  8460 rebalancer_tool.cc:1074] tablet a398da35f18c4df394db82c5bf89aeb1: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '' move scheduled
I20240502 01:56:25.740582  8460 rebalancer_tool.cc:1074] tablet ca458907378e42d48adaa0ee647db4cd: '218bc8c1f9ce46059898fc738179adf9' -> '' move scheduled
I20240502 01:56:25.783221  8460 rebalancer_tool.cc:1074] tablet 9a5f486852554e2fa2d07f7b9d686ced: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '' move scheduled
I20240502 01:56:27.661044  8460 rebalancer_tool.cc:1136] tablet ca458907378e42d48adaa0ee647db4cd: '218bc8c1f9ce46059898fc738179adf9' -> '' move completed: OK
I20240502 01:56:27.770313  8460 rebalancer_tool.cc:1136] tablet a398da35f18c4df394db82c5bf89aeb1: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '' move completed: OK
I20240502 01:56:27.833174  8460 rebalancer_tool.cc:1136] tablet ac3325a06d8841718158e1f5f181a398: '07cdb213c77c4ffda1f999b66b725399' -> '' move completed: OK
I20240502 01:56:27.924698  8460 rebalancer_tool.cc:1136] tablet 9a5f486852554e2fa2d07f7b9d686ced: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '' move completed: OK
Illegal state: tablet server 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): unacceptable health status UNAVAILABLE

W20240502 01:56:30.056079  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:56:30.073935  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:30.091022  8385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 01:56:30.102427  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:56:30.113575  8337 consensus_peers.cc:485] T 265a9645e1f141cb88c786b9aef8f90e P 6d34e50fb5dd4a1db58013d4863b8992 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:56:30.124742  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:30.130415  6447 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:30.137157  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:56:30.186030  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:56:30.190315  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:30.222337  8385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:30.223054  6316 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:56:30.237563  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:30.243570  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:30.284222  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:30.316663  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:30.333386  6316 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 01:56:30.409462  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:56:30.491216  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:56:30.497521  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:56:30.498795  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:56:30.499614  6447 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:30.561029  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:30.574910  6335 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "866f87ea9f7f48edb667b1b2eb3fa239"
dest_uuid: "94ffd7aaf85b4e17881241b8396fa2f3"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:38828
I20240502 01:56:30.575145  6335 raft_consensus.cc:595] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:56:30.588399  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:56:30.612051  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:56:30.614753 16283 raft_consensus.cc:979] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3: : Instructing follower 07cdb213c77c4ffda1f999b66b725399 to start an election
I20240502 01:56:30.614871 16283 raft_consensus.cc:1067] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Signalling peer 07cdb213c77c4ffda1f999b66b725399 to start an election
I20240502 01:56:30.615723  6469 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "866f87ea9f7f48edb667b1b2eb3fa239"
dest_uuid: "07cdb213c77c4ffda1f999b66b725399"
 from {username='slave'} at 127.25.109.131:39171
I20240502 01:56:30.615981  6469 raft_consensus.cc:482] T 866f87ea9f7f48edb667b1b2eb3fa239 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:56:30.616076  6469 raft_consensus.cc:3037] T 866f87ea9f7f48edb667b1b2eb3fa239 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:56:30.618889  6469 raft_consensus.cc:504] T 866f87ea9f7f48edb667b1b2eb3fa239 P 07cdb213c77c4ffda1f999b66b725399 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:56:30.620497  6335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "866f87ea9f7f48edb667b1b2eb3fa239" candidate_uuid: "07cdb213c77c4ffda1f999b66b725399" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "94ffd7aaf85b4e17881241b8396fa2f3"
I20240502 01:56:30.620728  6335 raft_consensus.cc:3032] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:56:30.620791  6335 raft_consensus.cc:729] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 94ffd7aaf85b4e17881241b8396fa2f3, State: Running, Role: LEADER
I20240502 01:56:30.621008  6335 consensus_queue.cc:260] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:56:30.621361  6335 raft_consensus.cc:3037] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:56:30.623656  6469 leader_election.cc:288] T 866f87ea9f7f48edb667b1b2eb3fa239 P 07cdb213c77c4ffda1f999b66b725399 [CANDIDATE]: Term 2 election: Requested vote from peers 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245), 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135:40543)
I20240502 01:56:30.624032  8269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "866f87ea9f7f48edb667b1b2eb3fa239" candidate_uuid: "07cdb213c77c4ffda1f999b66b725399" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "436822a397664c54b8d1b9eb3bc6cff4"
I20240502 01:56:30.624207  8269 raft_consensus.cc:3037] T 866f87ea9f7f48edb667b1b2eb3fa239 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:56:30.624308  6335 raft_consensus.cc:2454] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07cdb213c77c4ffda1f999b66b725399 in term 2.
I20240502 01:56:30.631170  6447 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:56:30.643172  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:30.646596  6402 leader_election.cc:302] T 866f87ea9f7f48edb667b1b2eb3fa239 P 07cdb213c77c4ffda1f999b66b725399 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07cdb213c77c4ffda1f999b66b725399, 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 
I20240502 01:56:30.648078 16442 raft_consensus.cc:2781] T 866f87ea9f7f48edb667b1b2eb3fa239 P 07cdb213c77c4ffda1f999b66b725399 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:56:30.648212 16442 raft_consensus.cc:686] T 866f87ea9f7f48edb667b1b2eb3fa239 P 07cdb213c77c4ffda1f999b66b725399 [term 2 LEADER]: Becoming Leader. State: Replica: 07cdb213c77c4ffda1f999b66b725399, State: Running, Role: LEADER
I20240502 01:56:30.648478 16442 consensus_queue.cc:237] T 866f87ea9f7f48edb667b1b2eb3fa239 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:56:30.656538  8269 raft_consensus.cc:2454] T 866f87ea9f7f48edb667b1b2eb3fa239 P 436822a397664c54b8d1b9eb3bc6cff4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07cdb213c77c4ffda1f999b66b725399 in term 2.
I20240502 01:56:30.660421  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:56:30.660444  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:56:30.692891  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:56:30.695037  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:56:30.697427  5915 catalog_manager.cc:5484] T 866f87ea9f7f48edb667b1b2eb3fa239 P 07cdb213c77c4ffda1f999b66b725399 reported cstate change: term changed from 1 to 2, leader changed from 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) to 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132). New cstate: current_term: 2 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
W20240502 01:56:30.734658  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:56:30.782222  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:30.785770  8385 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:30.786233  8384 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:30.806408  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:30.812738  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:30.844092  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:56:30.932979  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:31.053448  6335 raft_consensus.cc:1261] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:56:31.054145 16442 consensus_queue.cc:1035] T 866f87ea9f7f48edb667b1b2eb3fa239 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:56:31.090090  8270 raft_consensus.cc:1261] T 866f87ea9f7f48edb667b1b2eb3fa239 P 436822a397664c54b8d1b9eb3bc6cff4 [term 2 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:56:31.090960 16132 consensus_queue.cc:1035] T 866f87ea9f7f48edb667b1b2eb3fa239 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:56:31.222770  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:31.251392  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:31.317929  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:31.328065  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:31.392846  8384 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:31.431238  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:31.438943  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 01:56:31.548692  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:31.616307  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:31.628990  8384 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:31.667418  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:31.668354  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:31.717355  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:31.719650  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:31.720033  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:31.720908  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:31.771854  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:56:31.790735  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:31.818668  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:31.823748  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:31.854590  8384 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:31.893347  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:31.913113  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 01:56:31.942189  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:32.017984  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:32.073935  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:32.093605  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:32.105990  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:32.139679  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:32.143093  8384 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:32.158222  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 01:56:32.158293  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:56:32.235922  6469 consensus_queue.cc:237] T 866f87ea9f7f48edb667b1b2eb3fa239 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:56:32.241189  8270 raft_consensus.cc:1261] T 866f87ea9f7f48edb667b1b2eb3fa239 P 436822a397664c54b8d1b9eb3bc6cff4 [term 2 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:56:32.243119 16657 consensus_queue.cc:1035] T 866f87ea9f7f48edb667b1b2eb3fa239 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:56:32.257100 16442 raft_consensus.cc:2932] T 866f87ea9f7f48edb667b1b2eb3fa239 P 07cdb213c77c4ffda1f999b66b725399 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:32.257905  8270 raft_consensus.cc:2932] T 866f87ea9f7f48edb667b1b2eb3fa239 P 436822a397664c54b8d1b9eb3bc6cff4 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:32.264101  5916 catalog_manager.cc:5484] T 866f87ea9f7f48edb667b1b2eb3fa239 P 436822a397664c54b8d1b9eb3bc6cff4 reported cstate change: config changed from index 5 to 7, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New cstate: current_term: 2 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:32.279290  6316 tablet_service.cc:1514] Processing DeleteTablet for tablet 866f87ea9f7f48edb667b1b2eb3fa239 with delete_type TABLET_DATA_TOMBSTONED (TS 94ffd7aaf85b4e17881241b8396fa2f3 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:41134
I20240502 01:56:32.279237  6469 consensus_queue.cc:237] T 866f87ea9f7f48edb667b1b2eb3fa239 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
W20240502 01:56:32.288967  6402 consensus_peers.cc:485] T 866f87ea9f7f48edb667b1b2eb3fa239 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133:43379): Couldn't send request to peer 60298ffea1dd4c25a38765c2ec9ec9d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 866f87ea9f7f48edb667b1b2eb3fa239. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:56:32.290911  8270 raft_consensus.cc:1261] T 866f87ea9f7f48edb667b1b2eb3fa239 P 436822a397664c54b8d1b9eb3bc6cff4 [term 2 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 01:56:32.291553 16657 consensus_queue.cc:1035] T 866f87ea9f7f48edb667b1b2eb3fa239 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:56:32.295382 16657 raft_consensus.cc:2932] T 866f87ea9f7f48edb667b1b2eb3fa239 P 07cdb213c77c4ffda1f999b66b725399 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:56:32.299914  8270 raft_consensus.cc:2932] T 866f87ea9f7f48edb667b1b2eb3fa239 P 436822a397664c54b8d1b9eb3bc6cff4 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:56:32.303014 16741 tablet_replica.cc:330] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3: stopping tablet replica
I20240502 01:56:32.303299 16741 raft_consensus.cc:2229] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:32.303558 16741 raft_consensus.cc:2258] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:32.305176 16741 ts_tablet_manager.cc:1875] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:56:32.311439  5905 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 866f87ea9f7f48edb667b1b2eb3fa239 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 01:56:32.313575  5916 catalog_manager.cc:5484] T 866f87ea9f7f48edb667b1b2eb3fa239 P 07cdb213c77c4ffda1f999b66b725399 reported cstate change: config changed from index 7 to 8, VOTER 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133) added. New cstate: current_term: 2 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:56:32.314133 16741 ts_tablet_manager.cc:1888] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 01:56:32.314303 16741 log.cc:1196] T 866f87ea9f7f48edb667b1b2eb3fa239 P 94ffd7aaf85b4e17881241b8396fa2f3: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/wals/866f87ea9f7f48edb667b1b2eb3fa239
I20240502 01:56:32.316238  5907 catalog_manager.cc:4830] TS 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245): tablet 866f87ea9f7f48edb667b1b2eb3fa239 (table rebalance_test_table_3 [id=63a163be603345428cc84c86b96467f0]) successfully deleted
W20240502 01:56:32.379156  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:56:32.420929  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:32.421211  8337 consensus_peers.cc:485] T d8a6db6ae11e4cb8881aee95bdb0e59d P 6d34e50fb5dd4a1db58013d4863b8992 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:32.437641  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:56:32.496676  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:32.508139  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:32.508793  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:32.520725  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:56:32.570760  8337 consensus_peers.cc:485] T 265a9645e1f141cb88c786b9aef8f90e P 6d34e50fb5dd4a1db58013d4863b8992 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:56:32.608394  8384 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 01:56:32.617782  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:56:32.627671  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:32.642159  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:32.705075  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:32.714069  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:32.714607  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:32.737968  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 01:56:32.748790  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:32.770277  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:32.794591 16845 ts_tablet_manager.cc:902] T 866f87ea9f7f48edb667b1b2eb3fa239 P 60298ffea1dd4c25a38765c2ec9ec9d5: Initiating tablet copy from peer 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867)
I20240502 01:56:32.810526  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:32.844571  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:32.845009  6000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240502 01:56:32.845713  8384 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 01:56:32.846561  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:56:32.914623 16845 tablet_copy_client.cc:308] T 866f87ea9f7f48edb667b1b2eb3fa239 P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.132:36867
I20240502 01:56:32.916113  6489 tablet_copy_service.cc:140] P 07cdb213c77c4ffda1f999b66b725399: Received BeginTabletCopySession request for tablet 866f87ea9f7f48edb667b1b2eb3fa239 from peer 60298ffea1dd4c25a38765c2ec9ec9d5 ({username='slave'} at 127.25.109.133:42089)
I20240502 01:56:32.916272  6489 tablet_copy_service.cc:161] P 07cdb213c77c4ffda1f999b66b725399: Beginning new tablet copy session on tablet 866f87ea9f7f48edb667b1b2eb3fa239 from peer 60298ffea1dd4c25a38765c2ec9ec9d5 at {username='slave'} at 127.25.109.133:42089: session id = 60298ffea1dd4c25a38765c2ec9ec9d5-866f87ea9f7f48edb667b1b2eb3fa239
I20240502 01:56:32.919252  6489 tablet_copy_source_session.cc:205] T 866f87ea9f7f48edb667b1b2eb3fa239 P 07cdb213c77c4ffda1f999b66b725399: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:56:32.920737 16845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 866f87ea9f7f48edb667b1b2eb3fa239. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 01:56:32.923888  6136 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 01:56:32.925179  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:56:32.971949 16845 tablet_copy_client.cc:783] T 866f87ea9f7f48edb667b1b2eb3fa239 P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Starting download of 0 data blocks...
I20240502 01:56:32.972301 16845 tablet_copy_client.cc:647] T 866f87ea9f7f48edb667b1b2eb3fa239 P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Starting download of 1 WAL segments...
I20240502 01:56:32.978269 16845 tablet_copy_client.cc:523] T 866f87ea9f7f48edb667b1b2eb3fa239 P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:56:32.982555 16845 tablet_bootstrap.cc:492] T 866f87ea9f7f48edb667b1b2eb3fa239 P 60298ffea1dd4c25a38765c2ec9ec9d5: Bootstrap starting.
W20240502 01:56:32.986927  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:32.989463  6402 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 01:56:33.069538  6270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 01:56:33.107327  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:33.129179  8337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 01:56:33.167363 16845 tablet_bootstrap.cc:492] T 866f87ea9f7f48edb667b1b2eb3fa239 P 60298ffea1dd4c25a38765c2ec9ec9d5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:56:33.167912 16845 tablet_bootstrap.cc:492] T 866f87ea9f7f48edb667b1b2eb3fa239 P 60298ffea1dd4c25a38765c2ec9ec9d5: Bootstrap complete.
I20240502 01:56:33.168183 16845 ts_tablet_manager.cc:1367] T 866f87ea9f7f48edb667b1b2eb3fa239 P 60298ffea1dd4c25a38765c2ec9ec9d5: Time spent bootstrapping tablet: real 0.186s	user 0.060s	sys 0.008s
I20240502 01:56:33.168889 16845 raft_consensus.cc:348] T 866f87ea9f7f48edb667b1b2eb3fa239 P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 01:56:33.169088 16845 raft_consensus.cc:729] T 866f87ea9f7f48edb667b1b2eb3fa239 P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60298ffea1dd4c25a38765c2ec9ec9d5, State: Initialized, Role: FOLLOWER
I20240502 01:56:33.169303 16845 consensus_queue.cc:260] T 866f87ea9f7f48edb667b1b2eb3fa239 P 60298ffea1dd4c25a38765c2ec9ec9d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 01:56:33.187126 16845 ts_tablet_manager.cc:1398] T 866f87ea9f7f48edb667b1b2eb3fa239 P 60298ffea1dd4c25a38765c2ec9ec9d5: Time spent starting tablet: real 0.019s	user 0.015s	sys 0.001s
I20240502 01:56:33.209434  6489 tablet_copy_service.cc:342] P 07cdb213c77c4ffda1f999b66b725399: Request end of tablet copy session 60298ffea1dd4c25a38765c2ec9ec9d5-866f87ea9f7f48edb667b1b2eb3fa239 received from {username='slave'} at 127.25.109.133:42089
I20240502 01:56:33.209601  6489 tablet_copy_service.cc:433] P 07cdb213c77c4ffda1f999b66b725399: ending tablet copy session 60298ffea1dd4c25a38765c2ec9ec9d5-866f87ea9f7f48edb667b1b2eb3fa239 on tablet 866f87ea9f7f48edb667b1b2eb3fa239 with peer 60298ffea1dd4c25a38765c2ec9ec9d5
W20240502 01:56:33.217377  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:33.219633  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:33.256906  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:33.273600  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:33.274410  7998 raft_consensus.cc:1203] T 866f87ea9f7f48edb667b1b2eb3fa239 P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8]   Dedup: 2.8->[]
I20240502 01:56:33.297847  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:33.300956  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:33.313504  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:33.428225  8384 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:33.437844  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:33.438553  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:33.443610  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:33.466784  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:33.479789  8384 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:33.530617  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:33.531467  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:33.533241  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:33.564770  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:33.871572  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:33.873236  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:56:33.875154  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:33.877249  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:33.944032  8384 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:33.950575  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:33.984143  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:34.065630  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:56:34.069597  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:34.097745  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:34.115152  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:34.125617  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:34.136674  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:34.137136  8384 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:34.166167  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:34.201890  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:34.309799  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:34.329437  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:34.388028  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:56:34.429201  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:34.450125  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:56:34.453198  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:56:34.480850  8384 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:34.507277  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:34.534649  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:34.751567  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:34.753024  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:34.762905  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:34.788817  8384 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:34.791877  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:34.799645  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:34.805862  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:34.806885  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:34.833251  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:34.841097  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:34.843919  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:56:34.897709  8337 consensus_peers.cc:485] T d8a6db6ae11e4cb8881aee95bdb0e59d P 6d34e50fb5dd4a1db58013d4863b8992 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:56:34.917734  8384 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:34.923696  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:34.948385  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:34.972699  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 01:56:34.980742  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:56:35.165095  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:56:35.172477  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:56:35.177735  8337 consensus_peers.cc:485] T 265a9645e1f141cb88c786b9aef8f90e P 6d34e50fb5dd4a1db58013d4863b8992 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:56:35.244562  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:56:35.246052  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:35.276188  8384 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 01:56:35.281175  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:56:35.282196  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:35.289852  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:35.294327  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:56:35.298288  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:56:35.321594  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:56:35.328610  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:35.343731  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:56:35.359491  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 01:56:35.431397  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:35.545328  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:35.598117  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:56:35.618710  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:35.624871  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:35.627066  8384 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:35.642248  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:35.650903  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:35.655812  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 01:56:35.780381  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:56:35.800383  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:35.819343  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:35.831854  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:35.894089  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:35.925657  8384 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:35.926797  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:35.987217  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:36.117437  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:36.124732  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:36.159953  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:36.188324  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:36.192114  8384 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:36.195765  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:36.206609  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:36.236150  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 01:56:36.260246  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:36.413714  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:36.439646  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:36.466624  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:36.468475  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:36.480412  8384 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:36.511360  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:36.526252  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:56:36.551383  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:36.690672  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:36.715736  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:36.759516  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:36.780846  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:36.786598  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:36.798270  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:36.799671  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:36.818006  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:36.838788  8384 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:36.856705  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:36.881323  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:36.888434  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:36.941943  8384 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:36.967986  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:36.992168  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 01:56:37.036530  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:56:37.100021  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:37.102087  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:37.129935  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:37.170253  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:37.187826  8384 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:37.230603  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:37.231283  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 01:56:37.267346  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:56:37.382191  8337 consensus_peers.cc:485] T d8a6db6ae11e4cb8881aee95bdb0e59d P 6d34e50fb5dd4a1db58013d4863b8992 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:37.405332  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:56:37.408339  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:37.454416  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:37.464182  8384 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:37.485980  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:37.509042  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:37.514604  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:37.601529  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:37.626399  8337 consensus_peers.cc:485] T 265a9645e1f141cb88c786b9aef8f90e P 6d34e50fb5dd4a1db58013d4863b8992 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:56:37.684973  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:37.686607  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:37.688094  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:37.695331  8384 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:37.707327  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:37.713407  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:37.736591  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:37.746312  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:56:37.750743  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:37.762945  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:37.836505  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:37.879889  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:37.890573  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:37.891816  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:56:37.941509  6000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 01:56:37.943238  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:56:38.122160  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:38.163709  6402 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 01:56:38.176856  8384 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:38.178372  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:38.188987  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:38.194236  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:38.201421  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:38.214656  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:56:38.219723  6136 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240502 01:56:38.288225  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:38.308449  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:38.310254  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:38.329421  8384 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:38.360062  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:38.364814  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:38.375998  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:38.390319  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 01:56:38.393983  8337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 01:56:38.427572  6270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 01:56:38.429941  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:38.430212  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:56:38.554890  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:38.589395  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:38.599442  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:38.666522  8384 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:38.713599  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:38.719241  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:38.735306  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:38.858713  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:56:38.886711  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:38.889559  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:38.907841  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:38.936657  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:38.950486  8384 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:38.959108  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:38.980633  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:39.033455  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:39.104342  6469 consensus_queue.cc:237] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 01:56:39.106432  7998 raft_consensus.cc:1261] T 4486b413e5c14ec6953e4895ee5797cf P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:56:39.107111 17104 consensus_queue.cc:1035] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:56:39.116155 17773 raft_consensus.cc:2932] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:56:39.120055  7998 raft_consensus.cc:2932] T 4486b413e5c14ec6953e4895ee5797cf P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:56:39.134815  5916 catalog_manager.cc:5484] T 4486b413e5c14ec6953e4895ee5797cf P 60298ffea1dd4c25a38765c2ec9ec9d5 reported cstate change: config changed from index 5 to 6, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New cstate: current_term: 1 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 01:56:39.154531  6180 tablet_service.cc:1514] Processing DeleteTablet for tablet 4486b413e5c14ec6953e4895ee5797cf with delete_type TABLET_DATA_TOMBSTONED (TS a2c90ee440b44b708bccc2e2d5385cc0 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41258
I20240502 01:56:39.156847  6469 consensus_queue.cc:237] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
W20240502 01:56:39.164868  6401 consensus_peers.cc:485] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4486b413e5c14ec6953e4895ee5797cf. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:56:39.166196  7998 raft_consensus.cc:1261] T 4486b413e5c14ec6953e4895ee5797cf P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:56:39.167387 17104 consensus_queue.cc:1035] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:56:39.172906  7998 raft_consensus.cc:2932] T 4486b413e5c14ec6953e4895ee5797cf P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:39.177170 17104 raft_consensus.cc:2932] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:39.186717  5914 catalog_manager.cc:5484] T 4486b413e5c14ec6953e4895ee5797cf P 60298ffea1dd4c25a38765c2ec9ec9d5 reported cstate change: config changed from index 6 to 7, VOTER 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) added. New cstate: current_term: 1 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } }
I20240502 01:56:39.201206  5905 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 4486b413e5c14ec6953e4895ee5797cf with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 01:56:39.206223 17783 tablet_replica.cc:330] T 4486b413e5c14ec6953e4895ee5797cf P a2c90ee440b44b708bccc2e2d5385cc0: stopping tablet replica
I20240502 01:56:39.206570 17783 raft_consensus.cc:2229] T 4486b413e5c14ec6953e4895ee5797cf P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:56:39.206913 17783 raft_consensus.cc:2258] T 4486b413e5c14ec6953e4895ee5797cf P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:56:39.207914 17783 ts_tablet_manager.cc:1875] T 4486b413e5c14ec6953e4895ee5797cf P a2c90ee440b44b708bccc2e2d5385cc0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:56:39.222470 17783 ts_tablet_manager.cc:1888] T 4486b413e5c14ec6953e4895ee5797cf P a2c90ee440b44b708bccc2e2d5385cc0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:56:39.222699 17783 log.cc:1196] T 4486b413e5c14ec6953e4895ee5797cf P a2c90ee440b44b708bccc2e2d5385cc0: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal/wals/4486b413e5c14ec6953e4895ee5797cf
I20240502 01:56:39.223616  5908 catalog_manager.cc:4830] TS a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197): tablet 4486b413e5c14ec6953e4895ee5797cf (table rebalance_test_table_6 [id=5606c3da105b4b86868c6c51ecdd39aa]) successfully deleted
W20240502 01:56:39.427073  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:56:39.436004  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:39.443118  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:56:39.455395  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:39.487919  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:39.492645  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:39.571653  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:39.573405  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:39.579967  8384 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:39.732060  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:39.752099  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:39.772297  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:39.789534  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:39.793830  8384 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:56:39.816591 17902 ts_tablet_manager.cc:902] T 4486b413e5c14ec6953e4895ee5797cf P 6d34e50fb5dd4a1db58013d4863b8992: Initiating tablet copy from peer 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867)
F20240502 01:56:39.818846 17902 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-02b106ec57b94e44a29947a80a030a23.queue_time_us, tablet-download-pool-02b106ec57b94e44a29947a80a030a23.queue_time_us) (tablet-download-pool-02b106ec57b94e44a29947a80a030a23.run_cpu_time_us, tablet-download-pool-02b106ec57b94e44a29947a80a030a23.run_cpu_time_us) (tablet-download-pool-02b106ec57b94e44a29947a80a030a23.run_wall_time_us, tablet-download-pool-02b106ec57b94e44a29947a80a030a23.run_wall_time_us) (tablet-download-pool-086509f1ee0a4a81b2f8194b50dbe159.queue_time_us, tablet-download-pool-086509f1ee0a4a81b2f8194b50dbe159.queue_time_us) (tablet-download-pool-086509f1ee0a4a81b2f8194b50dbe159.run_cpu_time_us, tablet-download-pool-086509f1ee0a4a81b2f8194b50dbe159.run_cpu_time_us) (tablet-download-pool-086509f1ee0a4a81b2f8194b50dbe159.run_wall_time_us, tablet-download-pool-086509f1ee0a4a81b2f8194b50dbe159.run_wall_time_us) (tablet-download-pool-265a9645e1f141cb88c786b9aef8f90e.queue_time_us, tablet-download-pool-265a9645e1f141cb88c786b9aef8f90e.queue_time_us) (tablet-download-pool-265a9645e1f141cb88c786b9aef8f90e.run_cpu_time_us, tablet-download-pool-265a9645e1f141cb88c786b9aef8f90e.run_cpu_time_us) (tablet-download-pool-265a9645e1f141cb88c786b9aef8f90e.run_wall_time_us, tablet-download-pool-265a9645e1f141cb88c786b9aef8f90e.run_wall_time_us) (tablet-download-pool-2cefdfa8851147999c1425c606f36b1f.queue_time_us, tablet-download-pool-2cefdfa8851147999c1425c606f36b1f.queue_time_us) (tablet-download-pool-2cefdfa8851147999c1425c606f36b1f.run_cpu_time_us, tablet-download-pool-2cefdfa8851147999c1425c606f36b1f.run_cpu_time_us) (tablet-download-pool-2cefdfa8851147999c1425c606f36b1f.run_wall_time_us, tablet-download-pool-2cefdfa8851147999c1425c606f36b1f.run_wall_time_us) (tablet-download-pool-3a7698f6bd4349ae832a22c853d76fea.queue_time_us, tablet-download-pool-3a7698f6bd4349ae832a22c853d76fea.queue_time_us) (tablet-download-pool-3a7698f6bd4349ae832a22c853d76fea.run_cpu_time_us, tablet-download-pool-3a7698f6bd4349ae832a22c853d76fea.run_cpu_time_us) (tablet-download-pool-3a7698f6bd4349ae832a22c853d76fea.run_wall_time_us, tablet-download-pool-3a7698f6bd4349ae832a22c853d76fea.run_wall_time_us) (tablet-download-pool-4486b413e5c14ec6953e4895ee5797cf.queue_time_us, tablet-download-pool-4486b413e5c14ec6953e4895ee5797cf.queue_time_us) (tablet-download-pool-4eb91f5df851459b8ecf09cbe34fefa5.queue_time_us, tablet-download-pool-4eb91f5df851459b8ecf09cbe34fefa5.queue_time_us) (tablet-download-pool-4eb91f5df851459b8ecf09cbe34fefa5.run_cpu_time_us, tablet-download-pool-4eb91f5df851459b8ecf09cbe34fefa5.run_cpu_time_us) (tablet-download-pool-4eb91f5df851459b8ecf09cbe34fefa5.run_wall_time_us, tablet-download-pool-4eb91f5df851459b8ecf09cbe34fefa5.run_wall_time_us) (tablet-download-pool-4f4088f3523f4cd0afdf0de0f447c7af.queue_time_us, tablet-download-pool-4f4088f3523f4cd0afdf0de0f447c7af.queue_time_us) (tablet-download-pool-4f4088f3523f4cd0afdf0de0f447c7af.run_cpu_time_us, tablet-download-pool-4f4088f3523f4cd0afdf0de0f447c7af.run_cpu_time_us) (tablet-download-pool-4f4088f3523f4cd0afdf0de0f447c7af.run_wall_time_us, tablet-download-pool-4f4088f3523f4cd0afdf0de0f447c7af.run_wall_time_us) (tablet-download-pool-6984d638991442e3adaed1b24bee95e9.queue_time_us, tablet-download-pool-6984d638991442e3adaed1b24bee95e9.queue_time_us) (tablet-download-pool-6984d638991442e3adaed1b24bee95e9.run_cpu_time_us, tablet-download-pool-6984d638991442e3adaed1b24bee95e9.run_cpu_time_us) (tablet-download-pool-6984d638991442e3adaed1b24bee95e9.run_wall_time_us, tablet-download-pool-6984d638991442e3adaed1b24bee95e9.run_wall_time_us) (tablet-download-pool-73ed91df7ff04846991c95b30a86ea7e.queue_time_us, tablet-download-pool-73ed91df7ff04846991c95b30a86ea7e.queue_time_us) (tablet-download-pool-73ed91df7ff04846991c95b30a86ea7e.run_cpu_time_us, tablet-download-pool-73ed91df7ff04846991c95b30a86ea7e.run_cpu_time_us) (tablet-download-pool-73ed91df7ff04846991c95b30a86ea7e.run_wall_time_us, tablet-download-pool-73ed91df7ff04846991c95b30a86ea7e.run_wall_time_us) (tablet-download-pool-9a5f486852554e2fa2d07f7b9d686ced.queue_time_us, tablet-download-pool-9a5f486852554e2fa2d07f7b9d686ced.queue_time_us) (tablet-download-pool-9a5f486852554e2fa2d07f7b9d686ced.run_cpu_time_us, tablet-download-pool-9a5f486852554e2fa2d07f7b9d686ced.run_cpu_time_us) (tablet-download-pool-9a5f486852554e2fa2d07f7b9d686ced.run_wall_time_us, tablet-download-pool-9a5f486852554e2fa2d07f7b9d686ced.run_wall_time_us) (tablet-download-pool-c58d6b2401c047cd9af56a66c2e00105.queue_time_us, tablet-download-pool-c58d6b2401c047cd9af56a66c2e00105.queue_time_us) (tablet-download-pool-c58d6b2401c047cd9af56a66c2e00105.run_cpu_time_us, tablet-download-pool-c58d6b2401c047cd9af56a66c2e00105.run_cpu_time_us) (tablet-download-pool-c58d6b2401c047cd9af56a66c2e00105.run_wall_time_us, tablet-download-pool-c58d6b2401c047cd9af56a66c2e00105.run_wall_time_us) (tablet-download-pool-ca458907378e42d48adaa0ee647db4cd.queue_time_us, tablet-download-pool-ca458907378e42d48adaa0ee647db4cd.queue_time_us) (tablet-download-pool-ca458907378e42d48adaa0ee647db4cd.run_cpu_time_us, tablet-download-pool-ca458907378e42d48adaa0ee647db4cd.run_cpu_time_us) (tablet-download-pool-ca458907378e42d48adaa0ee647db4cd.run_wall_time_us, tablet-download-pool-ca458907378e42d48adaa0ee647db4cd.run_wall_time_us) (tablet-download-pool-d7922fd756344d14a336ff0c54afdf73.queue_time_us, tablet-download-pool-d7922fd756344d14a336ff0c54afdf73.queue_time_us) (tablet-download-pool-d7922fd756344d14a336ff0c54afdf73.run_cpu_time_us, tablet-download-pool-d7922fd756344d14a336ff0c54afdf73.run_cpu_time_us) (tablet-download-pool-d7922fd756344d14a336ff0c54afdf73.run_wall_time_us, tablet-download-pool-d7922fd756344d14a336ff0c54afdf73.run_wall_time_us) (tablet-download-pool-d8a6db6ae11e4cb8881aee95bdb0e59d.queue_time_us, tablet-download-pool-d8a6db6ae11e4cb8881aee95bdb0e59d.queue_time_us) (tablet-download-pool-d8a6db6ae11e4cb8881aee95bdb0e59d.run_cpu_time_us, tablet-download-pool-d8a6db6ae11e4cb8881aee95bdb0e59d.run_cpu_time_us) (tablet-download-pool-d8a6db6ae11e4cb8881aee95bdb0e59d.run_wall_time_us, tablet-download-pool-d8a6db6ae11e4cb8881aee95bdb0e59d.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
    @     0x7fdd445abdcd  google::LogMessage::Fail() at ??:0
    @     0x7fdd445afb93  google::LogMessage::SendToLog() at ??:0
    @     0x7fdd445ab7cc  google::LogMessage::Flush() at ??:0
    @     0x7fdd445acf59  google::LogMessageFatal::~LogMessageFatal() at ??:0
    @     0x7fdd472ec742  kudu::TraceMetrics::InternName() at ??:0
W20240502 01:56:39.849838  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
    @     0x7fdd472b8f8b  kudu::ThreadPool::ThreadPool() at ??:0
I20240502 01:56:39.857260  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
    @     0x7fdd472b2cd6  kudu::ThreadPoolBuilder::Build() at ??:0
    @     0x7fdd567058de  kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
W20240502 01:56:39.862797  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
    @     0x7fdd5671ddaf  kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240502 01:56:39.869917  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
    @     0x7fdd56848480  kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
    @     0x7fdd56880a8d  kudu::tserver::TabletCopyRunnable::Run() at ??:0
    @     0x7fdd56863981  kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
    @     0x7fdd5686370e  std::_Function_handler<>::_M_invoke() at ??:0
I20240502 01:56:39.887405  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
    @           0xa60bcd  std::function<>::operator()() at ??:?
    @     0x7fdd472bcb75  kudu::ThreadPool::DispatchThread() at ??:0
    @     0x7fdd472c019c  kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
    @     0x7fdd472c0019  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa60bcd  std::function<>::operator()() at ??:?
I20240502 01:56:39.911162  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
    @     0x7fdd4729b01e  kudu::Thread::SuperviseThread() at ??:0
    @     0x7fdd4a6206db  start_thread at ??:0
    @     0x7fdd4345e71f  clone at ??:0
I20240502 01:56:39.952309  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:39.993147  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:40.077651  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:40.110670  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:56:40.125360  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 01:56:40.160051  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:40.177106  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:40.184316  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:40.212066  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:56:40.241858  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:40.257230  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:40.269590  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:40.273265  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:56:40.293493  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:56:40.330204  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:40.386631  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 01:56:40.429678  6135 connection.cc:535] client connection to 127.25.109.136:44359 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:56:40.430424  5999 connection.cc:535] client connection to 127.25.109.136:44359 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:56:40.430832  6401 connection.cc:535] client connection to 127.25.109.136:44359 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:56:40.431356  6269 connection.cc:535] client connection to 127.25.109.136:44359 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:56:40.435495  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:40.435624  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:40.435956  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:40.438294  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:40.438427  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:56:40.438498  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:40.438556  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:40.438758  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:40.448055  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:40.448208  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:40.448278  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:40.463917  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:56:40.487202  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:56:40.573258  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:56:40.731570  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:40.755915  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:40.788065  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:40.815961  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:40.832064  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:56:40.860431  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:40.868863  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:40.890616  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:40.915427  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:40.928174  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:40.932691  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:56:40.938433  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:40.958571  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:40.971652  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:41.046046  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:41.050526  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:41.053282  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:41.080245  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:41.085764  6180 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:56:41.121093 17908 raft_consensus.cc:482] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 01:56:41.121279 17908 raft_consensus.cc:504] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:56:41.121991 17908 leader_election.cc:288] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 01:56:41.124969  5999 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:56:41.125453  6000 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:41.125572  6000 leader_election.cc:302] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:56:41.126469 17908 raft_consensus.cc:2726] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 01:56:41.130189  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:41.130954  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:41.176395  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:41.233875  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:56:41.305742 17840 raft_consensus.cc:482] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 01:56:41.305944 17840 raft_consensus.cc:504] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:56:41.306694 17840 leader_election.cc:288] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 01:56:41.308812  6269 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:56:41.317842  6270 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:41.318017  6270 leader_election.cc:302] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:56:41.318279 17840 raft_consensus.cc:2726] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 01:56:41.324729 17570 raft_consensus.cc:482] T 73ed91df7ff04846991c95b30a86ea7e P 436822a397664c54b8d1b9eb3bc6cff4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 01:56:41.324945 17570 raft_consensus.cc:504] T 73ed91df7ff04846991c95b30a86ea7e P 436822a397664c54b8d1b9eb3bc6cff4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:56:41.325990 17570 leader_election.cc:288] T 73ed91df7ff04846991c95b30a86ea7e P 436822a397664c54b8d1b9eb3bc6cff4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197), 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359)
I20240502 01:56:41.327415  6200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ed91df7ff04846991c95b30a86ea7e" candidate_uuid: "436822a397664c54b8d1b9eb3bc6cff4" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" is_pre_election: true
I20240502 01:56:41.340384  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:41.346220  8203 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:56:41.354477  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:56:41.358914  8203 leader_election.cc:334] T 73ed91df7ff04846991c95b30a86ea7e P 436822a397664c54b8d1b9eb3bc6cff4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:56:41.359520  8203 leader_election.cc:302] T 73ed91df7ff04846991c95b30a86ea7e P 436822a397664c54b8d1b9eb3bc6cff4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 436822a397664c54b8d1b9eb3bc6cff4; no voters: 6d34e50fb5dd4a1db58013d4863b8992, a2c90ee440b44b708bccc2e2d5385cc0
I20240502 01:56:41.360407 17570 raft_consensus.cc:2726] T 73ed91df7ff04846991c95b30a86ea7e P 436822a397664c54b8d1b9eb3bc6cff4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 01:56:41.362524 17762 raft_consensus.cc:482] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 01:56:41.362694 17762 raft_consensus.cc:504] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:56:41.363584 17762 leader_election.cc:288] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135:40543)
I20240502 01:56:41.364320  8268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ed91df7ff04846991c95b30a86ea7e" candidate_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "436822a397664c54b8d1b9eb3bc6cff4" is_pre_election: true
I20240502 01:56:41.364586  8268 raft_consensus.cc:2454] T 73ed91df7ff04846991c95b30a86ea7e P 436822a397664c54b8d1b9eb3bc6cff4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c90ee440b44b708bccc2e2d5385cc0 in term 2.
I20240502 01:56:41.365738  6134 leader_election.cc:302] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 436822a397664c54b8d1b9eb3bc6cff4, a2c90ee440b44b708bccc2e2d5385cc0; no voters: 
W20240502 01:56:41.367045  6135 leader_election.cc:334] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:56:41.367493 17762 raft_consensus.cc:2781] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 01:56:41.367571 17762 raft_consensus.cc:482] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 01:56:41.367630 17762 raft_consensus.cc:3037] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:56:41.370270 17762 raft_consensus.cc:504] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:56:41.372306  8268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ed91df7ff04846991c95b30a86ea7e" candidate_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "436822a397664c54b8d1b9eb3bc6cff4"
I20240502 01:56:41.372517  8268 raft_consensus.cc:3037] T 73ed91df7ff04846991c95b30a86ea7e P 436822a397664c54b8d1b9eb3bc6cff4 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:56:41.375231  8268 raft_consensus.cc:2454] T 73ed91df7ff04846991c95b30a86ea7e P 436822a397664c54b8d1b9eb3bc6cff4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c90ee440b44b708bccc2e2d5385cc0 in term 3.
I20240502 01:56:41.375727  6134 leader_election.cc:302] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 436822a397664c54b8d1b9eb3bc6cff4, a2c90ee440b44b708bccc2e2d5385cc0; no voters: 
W20240502 01:56:41.377153  6135 leader_election.cc:334] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:56:41.378700 17762 leader_election.cc:288] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 [CANDIDATE]: Term 3 election: Requested vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 436822a397664c54b8d1b9eb3bc6cff4 (127.25.109.135:40543)
I20240502 01:56:41.379065 17762 raft_consensus.cc:2781] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:56:41.379408 17762 raft_consensus.cc:686] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 [term 3 LEADER]: Becoming Leader. State: Replica: a2c90ee440b44b708bccc2e2d5385cc0, State: Running, Role: LEADER
I20240502 01:56:41.379845 17762 consensus_queue.cc:237] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:56:41.385051  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:41.385085  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:41.404947  5916 catalog_manager.cc:5484] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 reported cstate change: term changed from 2 to 3, leader changed from 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136) to a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130). New cstate: current_term: 3 leader_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 01:56:41.433513  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:56:41.455196  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:41.479995  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:41.569818  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:41.724843  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:41.753096  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:41.774922  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:41.777174  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:56:41.789886  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:56:41.826169  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:41.832820  8455 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240502 01:55:43.595695  8462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:43.596243  8462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:43.608124  8462 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:55:46.463135  8462 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 01:55:48.130877  8462 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0053d72412224a79be83b7e007b51934: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:55:48.131156  8462 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1ed4a88787d6401481245c552170a456: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:55:48.131258  8462 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fcde569cfdf747c2aaecac7d265b4b1f: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 60298ffea1dd4c25a38765c2ec9ec9d5
W20240502 01:55:48.131372  8462 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c2701645b8bc497aaf63e970157bdd80: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 60298ffea1dd4c25a38765c2ec9ec9d5
W20240502 01:55:48.131471  8462 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5606c3da105b4b86868c6c51ecdd39aa: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 60298ffea1dd4c25a38765c2ec9ec9d5
W20240502 01:55:48.131588  8462 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c02dbeef0b7241e78a12e4bd7ef79e47: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 60298ffea1dd4c25a38765c2ec9ec9d5
W20240502 01:55:48.131685  8462 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f406d4a60a5e40c485796d37ff4d76bc: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 60298ffea1dd4c25a38765c2ec9ec9d5
W20240502 01:55:48.131783  8462 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ef7ebedfbbff4054b074bf7a7622b632: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:55:48.131888  8462 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0053d72412224a79be83b7e007b51934: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:55:48.132006  8462 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ef6f7a2e933049e5982deec12e9a670a: could not find any suitable replica to move from server 218bc8c1f9ce46059898fc738179adf9 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:55:48.132094  8462 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 63a163be603345428cc84c86b96467f0: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 436822a397664c54b8d1b9eb3bc6cff4
W20240502 01:55:48.132303  8462 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5606c3da105b4b86868c6c51ecdd39aa: could not find any suitable replica to move from server 218bc8c1f9ce46059898fc738179adf9 to server 60298ffea1dd4c25a38765c2ec9ec9d5
W20240502 01:55:48.132390  8462 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c02dbeef0b7241e78a12e4bd7ef79e47: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:55:48.132496  8462 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f406d4a60a5e40c485796d37ff4d76bc: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 436822a397664c54b8d1b9eb3bc6cff4
I20240502 01:55:48.686137  8462 rebalancer_tool.cc:1074] tablet 0cecd1b14e474dee825fae040ce1b31e: '218bc8c1f9ce46059898fc738179adf9' -> '9db17f84d37944098cbeb9e0d8456efb' move scheduled
I20240502 01:55:49.429912  8462 rebalancer_tool.cc:1074] tablet e4d4a1bd80f44b668a2d504e47fb744a: '94ffd7aaf85b4e17881241b8396fa2f3' -> '436822a397664c54b8d1b9eb3bc6cff4' move scheduled
I20240502 01:55:49.889763  8462 rebalancer_tool.cc:1074] tablet ac3325a06d8841718158e1f5f181a398: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '60298ffea1dd4c25a38765c2ec9ec9d5' move scheduled
I20240502 01:55:50.690268  8462 rebalancer_tool.cc:1084] tablet d7922fd756344d14a336ff0c54afdf73: '07cdb213c77c4ffda1f999b66b725399' -> '6d34e50fb5dd4a1db58013d4863b8992' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }.
  Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 01:55:50.690451  8462 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:55:52.561064  8462 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c02dbeef0b7241e78a12e4bd7ef79e47: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 60298ffea1dd4c25a38765c2ec9ec9d5
W20240502 01:55:52.587095  8462 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0053d72412224a79be83b7e007b51934: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:55:52.587294  8462 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1ed4a88787d6401481245c552170a456: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 60298ffea1dd4c25a38765c2ec9ec9d5
W20240502 01:55:52.587404  8462 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5606c3da105b4b86868c6c51ecdd39aa: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:55:52.587514  8462 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c02dbeef0b7241e78a12e4bd7ef79e47: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:55:52.587596  8462 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fcde569cfdf747c2aaecac7d265b4b1f: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:55:52.590525  8462 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 63a163be603345428cc84c86b96467f0: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 436822a397664c54b8d1b9eb3bc6cff4
W20240502 01:55:52.590729  8462 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ef7ebedfbbff4054b074bf7a7622b632: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 60298ffea1dd4c25a38765c2ec9ec9d5
W20240502 01:55:52.590909  8462 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c2701645b8bc497aaf63e970157bdd80: could not find any suitable replica to move from server 218bc8c1f9ce46059898fc738179adf9 to server 436822a397664c54b8d1b9eb3bc6cff4
W20240502 01:55:52.591090  8462 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f406d4a60a5e40c485796d37ff4d76bc: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 436822a397664c54b8d1b9eb3bc6cff4
W20240502 01:55:52.591343  8462 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0053d72412224a79be83b7e007b51934: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 436822a397664c54b8d1b9eb3bc6cff4
W20240502 01:55:52.593835  8462 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5606c3da105b4b86868c6c51ecdd39aa: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:55:52.593991  8462 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c02dbeef0b7241e78a12e4bd7ef79e47: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 60298ffea1dd4c25a38765c2ec9ec9d5
W20240502 01:55:52.594139  8462 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1ed4a88787d6401481245c552170a456: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:55:52.594290  8462 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c2701645b8bc497aaf63e970157bdd80: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:55:52.868886  8462 rebalancer_tool.cc:1074] tablet 89f7cf12d85442749e7d5b259ebca43f: '94ffd7aaf85b4e17881241b8396fa2f3' -> '' move scheduled
I20240502 01:55:53.424551  8462 rebalancer_tool.cc:1074] tablet afa5561f8c0e4c0fa9bbbd5da61348c7: '07cdb213c77c4ffda1f999b66b725399' -> '436822a397664c54b8d1b9eb3bc6cff4' move scheduled
I20240502 01:55:53.930042  8462 rebalancer_tool.cc:1074] tablet ca458907378e42d48adaa0ee647db4cd: '218bc8c1f9ce46059898fc738179adf9' -> '6d34e50fb5dd4a1db58013d4863b8992' move scheduled
I20240502 01:55:54.329955  8462 rebalancer_tool.cc:1084] tablet 2cefdfa8851147999c1425c606f36b1f: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '60298ffea1dd4c25a38765c2ec9ec9d5' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }.
  Pending config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 01:55:54.330128  8462 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:55:56.764051  8462 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fcde569cfdf747c2aaecac7d265b4b1f: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:55:56.764199  8462 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c2701645b8bc497aaf63e970157bdd80: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:55:56.764411  8462 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ef6f7a2e933049e5982deec12e9a670a: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 60298ffea1dd4c25a38765c2ec9ec9d5
W20240502 01:55:56.764688  8462 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c02dbeef0b7241e78a12e4bd7ef79e47: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:55:56.764861  8462 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0053d72412224a79be83b7e007b51934: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 436822a397664c54b8d1b9eb3bc6cff4
W20240502 01:55:56.765604  8462 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ef7ebedfbbff4054b074bf7a7622b632: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 60298ffea1dd4c25a38765c2ec9ec9d5
I20240502 01:55:56.893419  8462 rebalancer_tool.cc:1074] tablet 5be45a68cff24a75a03b2fcd4b7851f9: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '' move scheduled
I20240502 01:55:57.187438  8462 rebalancer_tool.cc:1074] tablet 265a9645e1f141cb88c786b9aef8f90e: '07cdb213c77c4ffda1f999b66b725399' -> '6d34e50fb5dd4a1db58013d4863b8992' move scheduled
I20240502 01:55:57.644147  8462 rebalancer_tool.cc:1074] tablet 7f2519b0c6404799ad84d120b45a7ea4: '218bc8c1f9ce46059898fc738179adf9' -> '9db17f84d37944098cbeb9e0d8456efb' move scheduled
I20240502 01:55:57.893831  8462 rebalancer_tool.cc:1084] tablet 866f87ea9f7f48edb667b1b2eb3fa239: '94ffd7aaf85b4e17881241b8396fa2f3' -> '436822a397664c54b8d1b9eb3bc6cff4' move ignored: Illegal state: tablet 866f87ea9f7f48edb667b1b2eb3fa239: replica 436822a397664c54b8d1b9eb3bc6cff4 is already present
I20240502 01:55:57.894011  8462 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:55:58.871562 11015 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.25.109.190:37927, user_credentials={real_user=slave}} blocked reactor thread for 59441.6us
W20240502 01:55:59.670898  8462 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c02dbeef0b7241e78a12e4bd7ef79e47: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:55:59.671289  8462 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f406d4a60a5e40c485796d37ff4d76bc: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:55:59.671435  8462 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c2701645b8bc497aaf63e970157bdd80: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:55:59.671546  8462 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1ed4a88787d6401481245c552170a456: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 6d34e50fb5dd4a1db58013d4863b8992
I20240502 01:55:59.855790  8462 rebalancer_tool.cc:1074] tablet a398da35f18c4df394db82c5bf89aeb1: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '' move scheduled
I20240502 01:55:59.886438  8462 rebalancer_tool.cc:1074] tablet 0039dd405cfc4dcea5e7353169e3cd52: '94ffd7aaf85b4e17881241b8396fa2f3' -> '' move scheduled
I20240502 01:56:00.141000  8462 rebalancer_tool.cc:1074] tablet 899963a8260a42c4b73fb80eda2ecaa9: '07cdb213c77c4ffda1f999b66b725399' -> '' move scheduled
I20240502 01:56:02.558070  8462 rebalancer_tool.cc:1136] tablet a398da35f18c4df394db82c5bf89aeb1: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '' move completed: OK
I20240502 01:56:10.748273  8462 rebalancer_tool.cc:1136] tablet e4d4a1bd80f44b668a2d504e47fb744a: '94ffd7aaf85b4e17881241b8396fa2f3' -> '436822a397664c54b8d1b9eb3bc6cff4' move completed: OK
I20240502 01:56:12.741989  8462 rebalancer_tool.cc:1123] tablet 0cecd1b14e474dee825fae040ce1b31e: '218bc8c1f9ce46059898fc738179adf9' -> '9db17f84d37944098cbeb9e0d8456efb' move is abandoned: Not found: Server with UUID 218bc8c1f9ce46059898fc738179adf9 not a member of the config. RaftConfig: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 01:56:12.942294  8462 rebalancer_tool.cc:1136] tablet ac3325a06d8841718158e1f5f181a398: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '60298ffea1dd4c25a38765c2ec9ec9d5' move completed: OK
I20240502 01:56:18.263721  8462 rebalancer_tool.cc:1136] tablet 0039dd405cfc4dcea5e7353169e3cd52: '94ffd7aaf85b4e17881241b8396fa2f3' -> '' move completed: OK
I20240502 01:56:18.443710  8462 rebalancer_tool.cc:1136] tablet afa5561f8c0e4c0fa9bbbd5da61348c7: '07cdb213c77c4ffda1f999b66b725399' -> '436822a397664c54b8d1b9eb3bc6cff4' move completed: OK
I20240502 01:56:18.509965  8462 rebalancer_tool.cc:1074] tablet 4eb91f5df851459b8ecf09cbe34fefa5: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '' move scheduled
I20240502 01:56:18.614029  8462 rebalancer_tool.cc:1074] tablet f309b40931dd444091869119eab33bbc: '94ffd7aaf85b4e17881241b8396fa2f3' -> '' move scheduled
W20240502 01:56:19.999121 14312 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.25.109.135:40543, user_credentials={real_user=slave}} blocked reactor thread for 87785.3us
I20240502 01:56:21.145737  8462 rebalancer_tool.cc:1136] tablet 899963a8260a42c4b73fb80eda2ecaa9: '07cdb213c77c4ffda1f999b66b725399' -> '' move completed: OK
I20240502 01:56:21.402662  8462 rebalancer_tool.cc:1136] tablet 7f2519b0c6404799ad84d120b45a7ea4: '218bc8c1f9ce46059898fc738179adf9' -> '9db17f84d37944098cbeb9e0d8456efb' move completed: OK
W20240502 01:56:22.781049 14951 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 01:56:22.451984 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.134:42517 (local address 127.0.0.1:43614)
0502 01:56:22.452223 (+   239us) negotiation.cc:107] Waiting for socket to connect
0502 01:56:22.452241 (+    18us) client_negotiation.cc:174] Beginning negotiation
0502 01:56:22.452346 (+   105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 01:56:22.779647 (+327301us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":124}
I20240502 01:56:22.948249  8462 rebalancer_tool.cc:1136] tablet 265a9645e1f141cb88c786b9aef8f90e: '07cdb213c77c4ffda1f999b66b725399' -> '6d34e50fb5dd4a1db58013d4863b8992' move completed: OK
I20240502 01:56:23.011267  8462 rebalancer_tool.cc:1136] tablet 5be45a68cff24a75a03b2fcd4b7851f9: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '' move completed: OK
I20240502 01:56:23.137912  8462 rebalancer_tool.cc:1136] tablet ca458907378e42d48adaa0ee647db4cd: '218bc8c1f9ce46059898fc738179adf9' -> '6d34e50fb5dd4a1db58013d4863b8992' move completed: OK
I20240502 01:56:25.115341  8462 rebalancer_tool.cc:1074] tablet 086509f1ee0a4a81b2f8194b50dbe159: '07cdb213c77c4ffda1f999b66b725399' -> '' move scheduled
I20240502 01:56:25.514045  8462 rebalancer_tool.cc:1074] tablet 02b106ec57b94e44a29947a80a030a23: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '' move scheduled
I20240502 01:56:27.211269  8462 rebalancer_tool.cc:1136] tablet 086509f1ee0a4a81b2f8194b50dbe159: '07cdb213c77c4ffda1f999b66b725399' -> '' move completed: OK
I20240502 01:56:27.330644  8462 rebalancer_tool.cc:1136] tablet 4eb91f5df851459b8ecf09cbe34fefa5: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '' move completed: OK
I20240502 01:56:27.383317  8462 rebalancer_tool.cc:1136] tablet f309b40931dd444091869119eab33bbc: '94ffd7aaf85b4e17881241b8396fa2f3' -> '' move completed: OK
I20240502 01:56:27.452209  8462 rebalancer_tool.cc:1136] tablet 89f7cf12d85442749e7d5b259ebca43f: '94ffd7aaf85b4e17881241b8396fa2f3' -> '' move completed: OK
I20240502 01:56:27.515465  8462 rebalancer_tool.cc:1074] tablet 866f87ea9f7f48edb667b1b2eb3fa239: '94ffd7aaf85b4e17881241b8396fa2f3' -> '' move scheduled
I20240502 01:56:27.622867  8462 rebalancer_tool.cc:1074] tablet 3a7698f6bd4349ae832a22c853d76fea: '218bc8c1f9ce46059898fc738179adf9' -> '' move scheduled
I20240502 01:56:27.652915  8462 rebalancer_tool.cc:1074] tablet 9a5f486852554e2fa2d07f7b9d686ced: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '' move scheduled
I20240502 01:56:27.715051  8462 rebalancer_tool.cc:1074] tablet d8a6db6ae11e4cb8881aee95bdb0e59d: '94ffd7aaf85b4e17881241b8396fa2f3' -> '' move scheduled
I20240502 01:56:29.132542  8462 rebalancer_tool.cc:1136] tablet 9a5f486852554e2fa2d07f7b9d686ced: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '' move completed: OK
I20240502 01:56:29.294965  8462 rebalancer_tool.cc:1136] tablet 3a7698f6bd4349ae832a22c853d76fea: '218bc8c1f9ce46059898fc738179adf9' -> '' move completed: OK
I20240502 01:56:32.308835  8462 rebalancer_tool.cc:1136] tablet 02b106ec57b94e44a29947a80a030a23: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '' move completed: OK
I20240502 01:56:32.325367  8462 rebalancer_tool.cc:1074] tablet 55de74ae85e541a19008fd2fc2da3ffd: '218bc8c1f9ce46059898fc738179adf9' -> '' move scheduled
I20240502 01:56:32.364042  8462 rebalancer_tool.cc:1074] tablet 4486b413e5c14ec6953e4895ee5797cf: '218bc8c1f9ce46059898fc738179adf9' -> '' move scheduled
I20240502 01:56:32.381830  8462 rebalancer_tool.cc:1136] tablet 4486b413e5c14ec6953e4895ee5797cf: '218bc8c1f9ce46059898fc738179adf9' -> '' move completed: OK
I20240502 01:56:32.449419  8462 rebalancer_tool.cc:1136] tablet 55de74ae85e541a19008fd2fc2da3ffd: '218bc8c1f9ce46059898fc738179adf9' -> '' move completed: OK
I20240502 01:56:33.728704  8462 rebalancer_tool.cc:1136] tablet 866f87ea9f7f48edb667b1b2eb3fa239: '94ffd7aaf85b4e17881241b8396fa2f3' -> '' move completed: OK
W20240502 01:56:40.430768 17929 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 01:56:40.027042 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.136:44359 (local address 127.0.0.1:38478)
0502 01:56:40.045910 (+ 18868us) negotiation.cc:107] Waiting for socket to connect
0502 01:56:40.045929 (+    19us) client_negotiation.cc:174] Beginning negotiation
0502 01:56:40.046067 (+   138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 01:56:40.430470 (+384403us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":18751,"thread_start_us":18500,"threads_started":1}
I20240502 01:56:40.436815  8462 rebalancer_tool.cc:1123] tablet d8a6db6ae11e4cb8881aee95bdb0e59d: '94ffd7aaf85b4e17881241b8396fa2f3' -> '' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:56:40.437004  8462 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): unacceptable health status UNAVAILABLE

W20240502 01:56:41.840768  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:56:41.844807  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:41.864228  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:56:41.900328  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:41.928246  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:41.943745  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:41.998605  8268 raft_consensus.cc:1261] T 73ed91df7ff04846991c95b30a86ea7e P 436822a397664c54b8d1b9eb3bc6cff4 [term 3 FOLLOWER]: Refusing update from remote peer a2c90ee440b44b708bccc2e2d5385cc0: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240502 01:56:41.999392 17762 consensus_queue.cc:1035] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:56:42.026822  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:56:42.069475  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:42.123977  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:56:42.204259  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 01:56:42.254231  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:42.273619  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:42.290169  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:42.293932  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:56:42.303015  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 01:56:42.323673  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:56:42.328119  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:56:42.411327  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:56:42.442560  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:42.452592  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:42.469424  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:56:42.509240  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:42.515172  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:42.542104  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:42.557862  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:56:42.602046  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:56:42.613238  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:56:42.615765  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:56:42.637910  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:56:42.671116  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:56:42.677402  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:56:42.688423  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:56:42.721354  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 01:56:42.730738 17908 raft_consensus.cc:482] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 01:56:42.730926 17908 raft_consensus.cc:504] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:56:42.731616 17908 leader_election.cc:288] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 01:56:42.733508  6000 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:56:42.733667  5999 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:56:42.733791  5999 leader_election.cc:302] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:56:42.734180 17908 raft_consensus.cc:2726] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 01:56:42.743963  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:42.766419  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 01:56:42.787815  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:56:42.803751  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:42.818881  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 01:56:42.823055  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:42.828225  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:56:42.833581  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 01:56:42.839491  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:42.848503  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:56:42.888572  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 01:56:42.890983  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:56:42.895972  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:56:42.924225  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:56:42.951042  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 01:56:43.001976  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 01:56:43.040516  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:43.055104  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:43.073010  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:43.123687  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:56:43.152374  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:56:43.183547  6401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 64 similar messages]
W20240502 01:56:43.208565  6000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240502 01:56:43.242852  6136 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 44 similar messages]
I20240502 01:56:43.256012  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:43.259683  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:56:43.260465  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:43.269399  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:43.289098  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:43.334599  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:56:43.336798  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:56:43.352634  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:56:43.361320  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:43.374872  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:43.399741  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:56:43.512324  6269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240502 01:56:43.544806 18409 raft_consensus.cc:482] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 01:56:43.544982 18409 raft_consensus.cc:504] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:56:43.545630 18409 leader_election.cc:288] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 01:56:43.546928  6269 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:56:43.552244  6270 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:43.552372  6270 leader_election.cc:302] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:56:43.552590 18409 raft_consensus.cc:2726] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:56:43.569527  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 01:56:43.585631  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:43.598162  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:43.626330  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:43.626401  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:43.629133  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:56:43.650326  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:43.704032  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:43.745321  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:43.753840  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:56:43.765717  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:56:43.797919  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 01:56:43.815372  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:56:43.815873  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:43.816954  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:43.820977  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:43.981640  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:44.015331  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:44.031365  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:44.077414  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:44.079793  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:44.133019  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:56:44.210110  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:44.238885  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:56:44.243505  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:56:44.247684  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:44.266036  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:44.272096  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:44.287417  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:44.327531  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:56:44.409168  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 01:56:44.410579  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:44.421742  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:44.443499  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:44.480557  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:44.495236  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:44.496970  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:56:44.560842  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:44.642204  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:44.667697  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:44.688587  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:44.726882  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:44.730484  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:44.746950  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:44.750842  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:44.764556 18422 raft_consensus.cc:482] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 01:56:44.764748 18422 raft_consensus.cc:504] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:56:44.765498 18422 leader_election.cc:288] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 01:56:44.773589  5999 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:56:44.802187  6000 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:44.802341  6000 leader_election.cc:302] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:56:44.802608 18422 raft_consensus.cc:2726] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 01:56:44.816396  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:56:44.847442  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:56:44.942859  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:44.977020  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:44.993621  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:56:45.014097  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:56:45.032349  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:56:45.038962  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:56:45.044724  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:45.045557  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:45.051811  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:45.070721  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:56:45.095964  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:56:45.152171  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:45.175997  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:45.178678  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:56:45.180599  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 01:56:45.239501  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:45.247140  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:45.249553  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:56:45.255649  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:45.267027  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:56:45.275686  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:56:45.304915  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:56:45.305573  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:45.317597  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:45.323406  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:45.326198  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:56:45.338647  6268 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.132:36867, user_credentials={real_user=slave}} blocked reactor thread for 83318.8us
I20240502 01:56:45.346130  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:45.357833  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:45.359517  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:56:45.368938  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:56:45.407615  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 01:56:45.409070  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:45.430349  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:56:45.447685  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:56:45.462033  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:56:45.486240  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 01:56:45.580111  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:56:45.589323 18409 raft_consensus.cc:482] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 01:56:45.589519 18409 raft_consensus.cc:504] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:56:45.590216 18409 leader_election.cc:288] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 01:56:45.592299  6269 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:56:45.608021  6270 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:45.608161  6270 leader_election.cc:302] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:56:45.608388 18409 raft_consensus.cc:2726] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 01:56:45.663167  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:45.668704  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:45.678627  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:45.691300  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:45.708256  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:45.759543  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:45.780901  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:56:45.785848  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 01:56:45.888392  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:45.893728  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:56:45.909499  6402 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.133:43379, user_credentials={real_user=slave}} blocked reactor thread for 49476.3us
I20240502 01:56:45.929039  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:45.954501  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:45.964147  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:56:45.967640  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:56:46.001345  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:46.012979  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:56:46.111523  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:56:46.188345  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 01:56:46.206081  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:46.243039  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:56:46.266759  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 01:56:46.294062  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:46.312052  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:56:46.326454  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:56:46.333793  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:56:46.343063  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:46.522487  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:46.531716  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:46.573961  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:46.580277  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:46.602267  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:46.604701  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:56:46.616078  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:46.626801  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:46.627614  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:46.693213  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:46.720816  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:46.741765  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:46.804661  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:56:46.880059  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:56:46.956818  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:46.982403  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:56:47.013978  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 01:56:47.018111  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:56:47.029029  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:47.040448  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 01:56:47.041685  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:47.043900  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:47.058779  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:47.124843  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:47.154945  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:47.155402  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:47.181533  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:56:47.200467  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:56:47.203342  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:47.214859  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:47.309110 18409 raft_consensus.cc:482] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 01:56:47.309304 18409 raft_consensus.cc:504] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:56:47.310057 18409 leader_election.cc:288] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 01:56:47.312547  6269 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:56:47.322594  6270 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:47.322727  6270 leader_election.cc:302] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:56:47.322939 18409 raft_consensus.cc:2726] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 01:56:47.451575  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:47.473080  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:56:47.530486  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:47.562472  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:56:47.576704  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 01:56:47.585702  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:47.597831  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:47.600296  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:47.600687  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:47.601435  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:56:47.603299  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:56:47.624608  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:47.626580  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:56:47.640600  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:56:47.645632  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:56:47.646445  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:56:47.655622  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 01:56:47.698001  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:47.699569  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:56:47.732864  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:47.733021  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:56:47.737516  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:56:47.741520  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:47.743155  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:47.806394  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:56:47.872155  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 01:56:47.917618  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:47.933704  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:56:47.948369  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:56:47.955801  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:56:47.970016  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:56:47.971566  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:56:47.984614  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:48.003152  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:48.003257  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:56:48.004859  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:56:48.022010  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:56:48.054196  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:56:48.095312  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 01:56:48.148454  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:48.167827  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:48.175762  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:48.186177  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:56:48.198259  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:48.202994  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:48.216169  6401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 78 similar messages]
I20240502 01:56:48.245735  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:56:48.277439  6000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 01:56:48.329128  6135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240502 01:56:48.382104  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:48.401638  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:48.439384  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:48.497110  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:48.498560  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:56:48.506613  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:56:48.528659  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:56:48.541363  6269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 55 similar messages]
I20240502 01:56:48.691381  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:48.696017  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:48.698890  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:48.711333  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:48.740407  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:48.750116  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:56:48.813195  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:56:48.835311  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 01:56:48.885723  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:48.887316  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 01:56:48.906836  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:56:48.934882  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 01:56:48.940848  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:48.946220  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:48.949234  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:48.968097  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:56:49.113060  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:56:49.154225  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:49.168710  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:49.185011  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:49.202574  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:49.214534  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:56:49.218114  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:49.347728  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:49.362380  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:56:49.367012  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:49.379043  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:49.391673  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:49.392627  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:49.397640 18422 raft_consensus.cc:482] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 01:56:49.397800 18422 raft_consensus.cc:504] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:56:49.398458 18422 leader_election.cc:288] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 01:56:49.400738  5999 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:56:49.417266  6000 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:49.417496  6000 leader_election.cc:302] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:56:49.417781 18422 raft_consensus.cc:2726] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 01:56:49.459093  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:56:49.558390  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 01:56:49.622639  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:56:49.632941  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:49.653889  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 01:56:49.702230  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:49.707455  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:49.734995  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:49.787281  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:49.805281  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:49.811323  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:49.819856  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:56:49.946573  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:49.973577  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:49.989087  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:50.002269  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:50.004930  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:56:50.030079  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:56:50.090608  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:56:50.091832  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:56:50.141479  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:50.147269  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:56:50.155812  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:50.189834  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:50.198745  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:56:50.226840  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:50.229094  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 01:56:50.230640  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:50.246284  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:56:50.249475  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:50.264282  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:50.292954  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:50.293174  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:50.295884  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:50.358922  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:50.398254  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:56:50.419075  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:50.420203  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:50.424266  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:50.425832  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:50.466681  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:56:50.495676  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:50.496340  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:50.516829  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:56:50.519194  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:50.540706  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:50.548259  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:50.563262  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:50.610443  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:50.647189  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:56:50.652587  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:50.689188  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:50.720500  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:50.720650  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:50.756364  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:50.785341  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:56:50.864437  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 01:56:50.864616  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:56:50.910859  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:50.930785  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:50.948066  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:51.000160  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:56:51.002511  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:51.030076  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:56:51.056757  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:51.217419  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:51.218075  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:51.247009  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:51.268766  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:51.279700  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:56:51.285395  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:51.294348  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:56:51.300920  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:56:51.325857  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:51.336510  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:51.350497  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:56:51.368352  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:56:51.382570  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:56:51.428792  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:51.434067  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:51.451426  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:56:51.597950  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 01:56:51.738265  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:51.746343  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:51.786681  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:51.789084  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:51.800264  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:51.817183  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:51.833222  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:56:51.835331  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:51.838831  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:51.845155  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:56:51.913010  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:56:51.950316  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:56:51.955436  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:51.969714  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:56:51.987084  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 01:56:52.124055  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:52.141959  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:52.151470  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:52.179209  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:52.181236  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:52.227094  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:56:52.228596  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:52.253082  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:52.404287  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:56:52.449847  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:56:52.460135  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:52.479851  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:52.515046  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:52.528649  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:52.530592  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:52.565429  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:56:52.567970  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:56:52.568614  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:56:52.583961  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:56:52.586957  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:56:52.587970  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:52.598686  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:52.625388  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:56:52.654515  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:56:52.665213  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 01:56:52.676367  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:52.683827  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:56:52.708912  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:56:52.720453 19964 raft_consensus.cc:482] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 01:56:52.720677 19964 raft_consensus.cc:504] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:56:52.721447 19964 leader_election.cc:288] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 01:56:52.724479  6269 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:56:52.724808  6270 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:52.724907  6270 leader_election.cc:302] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:56:52.725783 19964 raft_consensus.cc:2726] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:56:52.736403  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:56:52.779999  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:56:52.815609  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:52.857966  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 01:56:52.937973  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:52.956462  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:56:52.976213  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:56:52.978216  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:52.981063  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:56:52.985038  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:52.995244  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:56:52.998790  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:53.005967  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:53.018187  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:53.032652  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:56:53.055040  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 01:56:53.097084  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:53.104190  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:53.107512  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:53.124226  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:56:53.131310  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:56:53.140640  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:53.164767  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:53.185166  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:53.242785  6402 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240502 01:56:53.330811  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:53.340997  6136 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 01:56:53.356477  5999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240502 01:56:53.362087  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:53.369210  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:53.369503  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:56:53.375664  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:53.393682  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:53.396195  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:56:53.495985  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:56:53.517292  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:53.579996  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:53.586416  6269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240502 01:56:53.593019  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:53.606451  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:53.641923  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:53.648900  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:56:53.654541  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:53.672269  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:56:53.743278  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:56:53.823091  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 01:56:53.857437  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:53.864593  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:56:53.898206  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:56:53.929518  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:53.931643  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:53.937532  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:53.969763  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:54.010807  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:56:54.025852  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 01:56:54.123265  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:54.139261  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:54.158331  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:54.198539  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:54.224653  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:54.239465  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:54.251040  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:54.268761  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:54.279819  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:54.285627  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:54.305037  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:56:54.315338 19630 raft_consensus.cc:482] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 01:56:54.315537 19630 raft_consensus.cc:504] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:56:54.316560 19630 leader_election.cc:288] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 01:56:54.319523  5999 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:56:54.319824  6000 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:54.319964  6000 leader_election.cc:302] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:56:54.320688 19630 raft_consensus.cc:2726] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 01:56:54.328162  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:56:54.400223  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:56:54.416646  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 01:56:54.454949  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:56:54.472699  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 01:56:54.581552  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:54.585835  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:54.599143  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:54.609402  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:54.627108  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:54.677263  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:56:54.716910  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 01:56:54.783339  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:54.792225  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:54.828771  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:54.829406  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:54.879391  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:54.887601  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:56:54.887732  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:56:54.928134  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:56:54.979005  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:54.996853  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:55.044930  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:55.062072  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:56:55.076687  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:56:55.087152  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:56:55.091758  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:55.097791  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:56:55.126958  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:56:55.140240  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:56:55.146334  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:55.243973  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:55.291981  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:55.292140  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:56:55.302415  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:55.313372  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:55.318521  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:56:55.332032  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:55.352555  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 01:56:55.358332  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:55.366637  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:55.383194  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 01:56:55.398614  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:56:55.464588  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:56:55.479725  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:56:55.526293  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:55.549477  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:55.553999  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:55.554849  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:55.557044  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:55.560545  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:55.582376  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:56:55.634889  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:55.655747  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 01:56:55.746031  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:55.766747  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:56:55.777415  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:56:55.781047  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:56:55.786665  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 01:56:55.804996  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:55.881405  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:56:55.892716  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:55.938264  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:55.964566  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:56:55.969272  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:56:56.060241  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 01:56:56.079828  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:56.086795  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:56.101766  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:56.111728  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:56:56.113965  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:56.129671  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:56.134379  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:56.145840  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:56.176590  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:56.188445  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:56.191949  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:56.220243  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:56:56.318662  6402 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.133:43379, user_credentials={real_user=slave}} blocked reactor thread for 67965.3us
W20240502 01:56:56.328226  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:56:56.352972  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:56.402667  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:56.510113  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:56.529806  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:56:56.532352  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:56.553846  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:56.569228  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:56.583698  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:56.584110  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:56.606622  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:56.620452  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:56.622467  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:56.657120  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:56.686767  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:56:56.700150  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:56:56.700352  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:56.702466  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 01:56:56.838128  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 01:56:56.922705  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:56.931178  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:56.944571  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:56:56.951038  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:56.960448  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:56.973842  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:56.974597  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:56:57.002822  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:57.005620  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:56:57.066183  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:57.074601  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:57.121531  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:56:57.159448  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:57.160007  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:57.332645  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:57.343983  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:57.354303  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:56:57.416780  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:56:57.419616  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:56:57.444234  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:57.447597  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:57.456102  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:56:57.475946  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:57.544270  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:56:57.626055  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 01:56:57.646338  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:57.672375  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:56:57.678763  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:56:57.681685  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:57.687007  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:57.697587  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:56:57.705531  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:56:57.708707  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:56:57.711057  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:57.732631  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:56:57.792907  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:56:57.823437  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:57.849630  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:56:57.854924  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:56:57.882617  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:57.892627  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:56:57.897353  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:57.902261  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 01:56:57.929597  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:57.929889  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:57.964097  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:56:57.999696  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:58.010668  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:58.020255  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:58.087491  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:58.159184  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:56:58.184799  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:58.185602  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:58.204931  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:56:58.205575  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:58.206266  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:58.206315  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:56:58.248020  6402 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240502 01:56:58.257501  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:58.310624  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:56:58.355108  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:58.359596  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:58.379664  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:58.391671  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:58.397071  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:56:58.421214  6136 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 01:56:58.430019  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:58.440474  6000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 72 similar messages]
I20240502 01:56:58.442238  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:56:58.537241  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:56:58.552629  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:56:58.605993  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 01:56:58.620468  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:58.640833  6269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240502 01:56:58.652602  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:56:58.654469  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:58.691929  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 01:56:58.699904  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:56:58.705618  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:58.737046  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:56:58.761088  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:58.793896  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:58.855036  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:58.883817  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:58.908004  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:58.917992  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:58.963611  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:59.006371  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:56:59.021559  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:59.033541  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:59.041581  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:59.068186  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:56:59.077431  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:56:59.084584  8204 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.133:43379, user_credentials={real_user=slave}} blocked reactor thread for 57867.4us
I20240502 01:56:59.090691  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:56:59.091781  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 01:56:59.100617  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:59.102497  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:56:59.113526  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 01:56:59.361007  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:59.388479  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:59.388602  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:59.392489  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:59.395354  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:59.418778  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:56:59.434870  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 01:56:59.437619  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:59.455116  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:56:59.485970  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:56:59.492535  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:56:59.496970  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:56:59.515839  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:59.517045  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:56:59.532637  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:56:59.604053  6268 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.132:36867, user_credentials={real_user=slave}} blocked reactor thread for 60617us
W20240502 01:56:59.733783  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 01:56:59.785440  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:56:59.786382  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:59.797941  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:56:59.804961  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:56:59.812789  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:56:59.818248  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:56:59.880147  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:00.000006  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 01:57:00.008044  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:00.024335  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:00.031937  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:00.042585  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:00.062655  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:00.093619  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:57:00.117553  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 01:57:00.120635  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:00.143477  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 01:57:00.148578  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:00.151906  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:57:00.187215  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:57:00.269110  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:00.348277  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:57:00.361140  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 01:57:00.392858  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:00.431077  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 01:57:00.439986  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:00.453452  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:00.498538  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:00.503355  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 01:57:00.506383  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:00.506820  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:00.526602  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:57:00.574088  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:57:00.610438  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 01:57:00.610910  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:00.650908  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:00.670708  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:00.680505  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:00.681793  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:00.684907  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 01:57:00.691116  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:00.697500  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:00.708935  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:00.717108  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:00.736582  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 01:57:00.743839  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:00.785187  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:00.799566  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:00.839004  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:00.861979  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:00.973654  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:57:01.011505  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:57:01.028370  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:57:01.042153  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 01:57:01.050976 21131 raft_consensus.cc:482] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 01:57:01.051131 21131 raft_consensus.cc:504] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:57:01.051748 21131 leader_election.cc:288] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 01:57:01.054072  6269 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:01.054394  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 01:57:01.066124  6270 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:01.066306  6270 leader_election.cc:302] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:57:01.066541 21131 raft_consensus.cc:2726] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:57:01.083521  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:01.084306  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:01.136464  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:01.159696  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:01.160468  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:01.163367  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:01.185560  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:01.278610  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:01.410878  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:01.415932  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:01.445307  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:01.457610  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:01.531308  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:01.558526  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 01:57:01.565296  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:01.638785  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:01.652863  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:01.684015  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:01.700670  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:01.704182  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:01.721463  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:01.732005  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 01:57:01.741014  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:01.816783 20792 raft_consensus.cc:482] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 01:57:01.816983 20792 raft_consensus.cc:504] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:57:01.817664 20792 leader_election.cc:288] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 01:57:01.820631  6000 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:01.820829  5999 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:01.820909  5999 leader_election.cc:302] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:57:01.821311 20792 raft_consensus.cc:2726] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:57:01.872651  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:01.885978  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:01.908928  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:01.946539  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:01.965912  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:01.994447  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:01.994632  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:02.032578  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:57:02.167440  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 01:57:02.168041  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:02.175493  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:02.196664  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:02.201459  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:02.269173  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:02.274608  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:02.297940  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:02.304221  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:02.326297  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:02.354001  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:02.376650  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:02.389643  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:02.403290  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:02.478029  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 01:57:02.503971  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:02.577428  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 01:57:02.668735  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:02.669198  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:02.671712  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:02.684226  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:02.699172  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 01:57:02.706157  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:02.735762  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:57:02.762185  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 01:57:02.771742  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:02.786983  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:57:02.806936  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:57:02.823551  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:02.852034  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 01:57:02.883108  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:02.887449  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 01:57:02.892737  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:02.899343  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:02.904194  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:02.908808  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:02.927230  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:02.972779  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:02.987845  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:57:03.050684  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 01:57:03.119626  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:03.143293  6268 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.132:36867, user_credentials={real_user=slave}} blocked reactor thread for 73784.9us
I20240502 01:57:03.172870  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:03.176231  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:03.191586  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:03.208272  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:03.270798  6401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 01:57:03.274200  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:03.279479  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:03.308648  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:57:03.309423  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:57:03.336262  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:57:03.348003  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 01:57:03.372444  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:03.376403  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:03.377236  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:03.390292  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:03.399565  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:03.432466  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:03.450661  6000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 01:57:03.455065  6135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 01:57:03.455935  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:57:03.464735  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:03.473136  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 01:57:03.636329  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 01:57:03.648932  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:03.658428  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:03.678033  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 01:57:03.679895  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:03.680300  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:03.692659  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:03.694761  6270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 01:57:03.699816  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 01:57:03.713766  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:03.730338  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:03.809855 21692 raft_consensus.cc:482] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 01:57:03.810052 21692 raft_consensus.cc:504] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:57:03.810698 21692 leader_election.cc:288] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 01:57:03.812043  6270 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:03.826210  6269 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:03.826409  6269 leader_election.cc:302] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:57:03.826804 21692 raft_consensus.cc:2726] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 01:57:03.920321  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:03.946580  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:03.988549  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:03.998809  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:04.005491  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:04.021394  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:04.041333 21479 raft_consensus.cc:482] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 01:57:04.041587 21479 raft_consensus.cc:504] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:57:04.042299 21479 leader_election.cc:288] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 01:57:04.045286  6000 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:04.045873  5999 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:04.045997  5999 leader_election.cc:302] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:57:04.046489 21479 raft_consensus.cc:2726] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 01:57:04.049594  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:04.055521  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:04.064560  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:04.078002  6270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.133:43379, user_credentials={real_user=slave}} blocked reactor thread for 94749.3us
I20240502 01:57:04.092532  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:04.101955  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:04.114317  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:04.115655  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 01:57:04.139325  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:04.163053  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:04.355345  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:04.387787  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:04.388124  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:04.401757  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:04.402170  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:04.413033  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:04.432309  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:57:04.519887  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:57:04.562265  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:04.625747  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 01:57:04.668519  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:04.712965  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:04.731767  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:04.733158  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:04.736275  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:04.772306  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:04.792950  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:04.793828  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:04.824069  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:04.832650  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:04.837631  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:04.857241  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:04.883082  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:04.985561  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:57:04.988270  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:05.084793  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 01:57:05.092880  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:05.111960  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 01:57:05.113790  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:05.114336  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:05.117669  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:05.150044  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:05.162403  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 01:57:05.180997  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:05.182312  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:05.266428  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:57:05.272711  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:57:05.318274  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 01:57:05.327792  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:05.334587  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 01:57:05.361748  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:05.363445  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 01:57:05.364524  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:05.383626  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:05.384341  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:05.407644  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:05.420401  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:57:05.527609  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:57:05.555689  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:05.641513  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 01:57:05.652681  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:05.655596  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:05.681517  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:05.695498  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 01:57:05.703492  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:05.711556  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:05.724714  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:05.734450  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:05.792639  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:57:05.812749  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:57:05.834767  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:57:05.864133  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 01:57:05.886049  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:05.921751  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:05.933089  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:05.947918  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:05.973810  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:05.995635  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:05.999457  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:06.044922  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:06.058559  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:06.059203  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:06.083395  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:06.101542  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:06.115206  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:06.116119  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:06.147310  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:57:06.265467  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:57:06.304152  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 01:57:06.374516  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:06.385712  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:06.412309  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:06.417598  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:06.424098  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:06.465937  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:06.472574  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:06.491724  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:06.504458  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:06.506177  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:06.524387  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:06.581926  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:06.599838  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:06.616070  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:06.706399  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 01:57:06.765998  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 01:57:06.794701  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:06.802279  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:06.815110  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:06.820601  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:06.836877  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:06.890827  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:06.892951  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:07.021067  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 01:57:07.022390  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:07.031519  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 01:57:07.040328  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:07.060696  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:07.061226  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:07.062603  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:07.068217  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:07.111773  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:07.224040  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:07.236755  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:07.278932  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:07.288552  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:07.299117  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:07.322976  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:07.374754  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:07.425094  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:57:07.560904  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:57:07.572428  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:57:07.585704  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 01:57:07.631368  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:07.642939  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:07.648581  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:07.660531  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:07.667066  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:07.674639  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:07.685678  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:07.696169  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:07.716575  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:07.717339  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:07.753054  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 01:57:07.762715  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:07.806406  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 01:57:07.831516  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:07.838616  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 01:57:07.844090  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:07.848843  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:57:07.885144  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:57:07.901638  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:57:07.909150  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:08.023607  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:08.055922  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:08.059708  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:08.089332  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:08.102089  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:08.129163  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:08.143500  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:57:08.143882  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 01:57:08.145915  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:08.168205  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:08.188506  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:08.197773  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:08.202138  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 01:57:08.204882  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:08.210989  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:57:08.235677  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 01:57:08.258375  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:08.287604  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:08.352897  6401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240502 01:57:08.370882  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 01:57:08.449957  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:08.452457  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:08.490378  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 01:57:08.501026  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:08.525184  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:08.528249  6000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240502 01:57:08.528877  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:08.545743  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:08.590960  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:08.660370  6135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 63 similar messages]
W20240502 01:57:08.666177  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:08.704578  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:08.721729  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:08.725636  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:08.739962  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:08.747548  6270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240502 01:57:08.751406  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:08.776516  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:08.805275  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:57:08.873888  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:08.912600  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:08.928455  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:08.928926  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:08.959012  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:08.971172  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:57:09.006328  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 01:57:09.009785  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:09.082414  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:09.146040  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:09.193346  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:09.197489  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 01:57:09.209062  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:09.214793  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:09.224332  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:09.241482  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:09.317286  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:09.372822  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:57:09.381139  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:09.412343  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:09.456470  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 01:57:09.468428  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:09.471349  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:09.472142  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:09.544121  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:09.544342  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:09.613782  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:57:09.656939  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:09.690920  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:09.712337  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:09.712448  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:09.712647  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:09.746487  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:09.750155  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 01:57:09.764006  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:09.841290  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:09.858711  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:09.897689  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:09.937822  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:09.946882  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:09.963797  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:10.005802  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:10.061856  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 01:57:10.109562  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:10.120676  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:10.125504  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:10.161571  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:57:10.175403  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:10.177065  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:10.177220  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:10.200593  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:10.215514  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:57:10.297613  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:57:10.398450  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:57:10.398491  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:57:10.411290  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 01:57:10.419335  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:10.420857  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:10.435520  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:10.436086  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:10.451685  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:10.462917  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 01:57:10.501989  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:10.512149  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:10.550890  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:10.569612  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:10.599222  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:10.599879  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:10.612041  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 01:57:10.632953  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:10.646903  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:10.647217  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:10.652256  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:57:10.767808  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:57:10.792310  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 01:57:10.881268  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:10.926179  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:10.945900  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:10.961220  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 01:57:10.968487  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:10.983229  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:10.983646  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:11.039088  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:11.042012  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:11.045750  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:11.069471  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:11.075449  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 01:57:11.075534  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:11.076455  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:11.078655  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:11.095670  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:57:11.114289  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:11.127940  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 01:57:11.285890 22766 raft_consensus.cc:482] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 01:57:11.286084 22766 raft_consensus.cc:504] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:57:11.286813 22766 leader_election.cc:288] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 01:57:11.289281  6269 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:11.289614  6270 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:11.289748  6270 leader_election.cc:302] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:57:11.290043 22766 raft_consensus.cc:2726] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:57:11.297609  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 01:57:11.353832  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:11.354193  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:11.354224  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:11.385099  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 01:57:11.388152  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:11.409409  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:11.438730  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:11.489607  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 01:57:11.575683  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:11.576117  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:11.592905  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:11.613090  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:11.617594  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 01:57:11.627699  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:11.655954  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:11.677678  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:11.732134  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:11.840445  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:11.857107  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:11.897962  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:11.916428  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:11.940172  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:11.972182  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:11.998319  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:12.114845  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 01:57:12.123680  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:12.123677  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:12.148875  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:12.149888  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 01:57:12.163589  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:12.165427  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:12.179327  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:57:12.208667  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:57:12.257349  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 01:57:12.260560  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:12.270397  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:12.290092  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:12.360165  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:12.362303  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:12.368608  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:12.379595  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:12.391099  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:12.408646  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:12.570706  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:57:12.653944  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 01:57:12.664528  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:12.680804  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:12.687402  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 01:57:12.691963  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:12.700852  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 01:57:12.703502  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:12.711244  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:12.717128  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:12.719987  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:12.724865  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:12.744678  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:12.755172  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:12.778834  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:57:12.798666  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:12.831853  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:12.835427  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 01:57:12.850100  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:12.853649  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:12.932478  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:57:13.004056  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 01:57:13.060240  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:13.067510  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 01:57:13.076265  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:13.076855  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:13.091832  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:13.094328  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:13.108647  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:13.242588  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:13.248332  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:13.288573  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 01:57:13.354676  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:13.364279  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:13.384231  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:13.406129  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:13.411731  6401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 79 similar messages]
I20240502 01:57:13.417929  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:13.422837  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:13.457607  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:13.461099  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:13.500208  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:13.534992  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:57:13.556877  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 01:57:13.558463  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:13.582275  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:57:13.590770  5999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240502 01:57:13.608997  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:13.613786  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:13.630316  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:13.662303  6135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 01:57:13.663566  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:57:13.764009  6270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240502 01:57:13.764936  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 01:57:13.775089  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:13.783185  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:13.797510  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:13.808714  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:13.820573  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:13.831166  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:13.832120  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:13.871111  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 01:57:13.967635  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:13.978750  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:57:13.980130  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 01:57:13.992159  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:13.998366  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:14.005424  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:14.047343  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:14.084254  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:14.124315  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:14.226843  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:14.242390  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 01:57:14.271183  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:14.280170  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:14.296396  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:14.300273  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:14.300665  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:14.301080  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:14.463199  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:57:14.497480  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 01:57:14.499938  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:14.567751  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:14.583499  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:14.584493  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:14.601188  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:14.605624  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:14.610275  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:14.642704  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:14.656033  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:57:14.658598  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 01:57:14.674981  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:14.689592  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:14.723508  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:14.726030  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:14.760529  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:14.799954  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:57:14.839381  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:14.969280  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:14.970477  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 01:57:14.973556  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:14.990084  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:14.994997  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:15.000650  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:15.014565  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 01:57:15.025393  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:15.039490  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:15.123899  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 01:57:15.150496  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:15.159418  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:15.176883  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:15.184067  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:15.189574  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:15.234890  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 01:57:15.245536  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:15.300663  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:57:15.343503  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:57:15.368487  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:15.384346  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:57:15.410668  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:15.468722  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 01:57:15.470289  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:15.491390  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:15.493084  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:15.519515  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:15.558239  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:15.570345  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:15.593813  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 01:57:15.739473  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:15.745028  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:15.752071  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:15.784518  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:15.827215  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:15.866892  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:15.877133  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:15.907554  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:15.932690  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:57:15.945267  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 01:57:15.952019  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:15.952766  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:15.958345  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:15.989568  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:15.994243  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:15.996752  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:16.080416  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:57:16.136606  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 01:57:16.138909  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:16.142534  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:16.170895  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:16.177356  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:16.212458  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:16.231379  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:16.292668  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:57:16.425706  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 01:57:16.437407  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:16.455186  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:16.459300  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:16.465582  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:16.488695  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 01:57:16.517164  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:16.523043  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:16.540452  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:16.544116  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:16.586797  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:16.589438  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 01:57:16.612694  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:16.660843  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:16.679214  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:16.701323  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 01:57:16.704727  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:16.705230  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:16.715548  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 01:57:16.747112  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:57:16.912348  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 01:57:16.946501  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:16.973007  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 01:57:16.975204  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:16.976032  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:16.976393  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 01:57:17.009398  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:17.012342  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:17.022907  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:17.062640  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:17.064136  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:17.065116  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:17.073549  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:17.081946  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:17.083144  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:17.130075  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 01:57:17.194249  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:17.199146  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:17.381057  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:17.390105  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 01:57:17.390822  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:17.415977  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:17.424073  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:17.426254  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:17.447077  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:17.465970  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:17.477727  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:17.492314  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:57:17.597755  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:57:17.610582  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 01:57:17.672260  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:17.690160  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:17.690774  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:17.702311  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:17.729480  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:17.758631  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:17.841595  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:17.888620  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:17.903962  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:17.933707  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:17.938994  6270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.133:43379, user_credentials={real_user=slave}} blocked reactor thread for 50468.8us
W20240502 01:57:17.947530  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 01:57:17.978245  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:17.990403  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:18.029525  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:18.043030  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:18.074211  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:57:18.074728  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:57:18.075116  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 01:57:18.248240  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:18.289424  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:18.292821  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:18.315630  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:18.336500  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 01:57:18.347695  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:18.353857  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:18.382452  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 01:57:18.383564  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:18.402029  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:18.429061  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:18.445694  6401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 01:57:18.447567  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:18.447911  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:18.451073  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:18.477402  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:18.494428  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 01:57:18.499835  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:18.508462  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:57:18.562539  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:57:18.700958  6135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 01:57:18.703617  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 01:57:18.759505  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:18.767601  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:18.775292  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:18.778395  5999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240502 01:57:18.811095  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:18.824018  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:18.833043  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:18.840087  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:18.841943  6270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 01:57:18.842916  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 01:57:18.887929  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:18.902048  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:18.914799  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 01:57:18.915179  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:18.925843  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:18.943261  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:18.967586  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:19.023221  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:57:19.063897  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:19.091390  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:57:19.117542  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 01:57:19.129159  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:19.141026  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:19.158430  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:19.166410  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:19.183347  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:19.188004  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:19.261956  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:57:19.331048  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 01:57:19.389163  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:19.440021  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:19.462644  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:19.465399  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:19.497048  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 01:57:19.502090  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:19.511211  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:19.511979  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:19.515102  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:19.537822  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 01:57:19.556677  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:19.572950  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:19.595091  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:19.595754  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 01:57:19.652345  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:19.657323  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:19.668639  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:19.769532  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:19.808451  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 01:57:19.870044  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:19.871560  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:19.884097  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:19.897586  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:19.901192  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:19.948467  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:20.054994  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 01:57:20.069324  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:20.083742  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:20.085886  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:20.087426  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:20.089875  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:20.117812  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:20.163511  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:20.172031  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 01:57:20.287389  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:20.323588  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:20.382165  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:20.383191  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:20.387993  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:20.388341  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 01:57:20.421321  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:20.452404  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 01:57:20.476652  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:20.579205  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 01:57:20.633158  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:20.637141  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:20.639938  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:20.645506  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 01:57:20.671173  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:20.672957  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:20.673236  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:20.692057  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:20.708663  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:20.709587  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 01:57:20.711854  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:20.712250  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:20.740144  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:20.751320  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:20.793810  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:20.794044  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:57:20.800242  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:20.890198  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:57:20.957612  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 01:57:20.989724  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:20.991189  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:21.004900  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:21.016932  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:21.028168  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 01:57:21.047685  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:21.048066  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:21.212157  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 01:57:21.219427  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:21.220196  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:21.220863  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:21.256258  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 01:57:21.272722  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:21.273723  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:21.283589  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:57:21.285266  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:21.299487  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:21.370404  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 01:57:21.388464  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:57:21.388778  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:21.477032  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:21.492604  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:21.505685  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:21.511215  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 01:57:21.530690  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:21.587563  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:21.602823  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:21.677819  5910 maintenance_manager.cc:419] P ee06754b44a64b83a28dea3c44bf4dfc: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033381
I20240502 01:57:21.727826  5909 maintenance_manager.cc:643] P ee06754b44a64b83a28dea3c44bf4dfc: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.049s	user 0.041s	sys 0.000s Metrics: {"bytes_written":20314,"cfile_init":1,"dirs.queue_time_us":389,"dirs.run_cpu_time_us":229,"dirs.run_wall_time_us":1714,"drs_written":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1144,"lbm_writes_lt_1ms":28,"peak_mem_usage":0,"rows_written":43,"thread_start_us":896,"threads_started":2,"wal-append.queue_time_us":296}
I20240502 01:57:21.729486  5910 maintenance_manager.cc:419] P ee06754b44a64b83a28dea3c44bf4dfc: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 6666 bytes on disk
I20240502 01:57:21.732268  5909 maintenance_manager.cc:643] P ee06754b44a64b83a28dea3c44bf4dfc: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.002s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":1881,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3}
W20240502 01:57:21.761513  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 01:57:21.782894  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:21.804591  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:21.817921  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:21.821197  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:21.837288  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 01:57:21.854401  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:21.856664  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:21.870970  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:21.899214  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:21.909606  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:21.909678  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 01:57:21.914359  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:21.934142  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:21.943310  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 01:57:21.948587  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:21.978936  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:21.985352  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 01:57:22.017019  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:22.164547  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:22.170135  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:22.188868  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:22.189152  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:22.200223  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:22.202111  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:22.271617  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:57:22.330415  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 01:57:22.387511 24090 raft_consensus.cc:482] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 01:57:22.387712 24090 raft_consensus.cc:504] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:57:22.388520 24090 leader_election.cc:288] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 01:57:22.391454  5999 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:22.405972  6000 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:22.406200  6000 leader_election.cc:302] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:57:22.411657 24090 raft_consensus.cc:2726] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 01:57:22.424858  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:22.426970  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:22.448284  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:22.448658  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:22.480456  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 01:57:22.481232  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:22.481374  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:22.539760  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:22.573164  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:22.628397  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:22.667011  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:22.670621  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:22.692734  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:22.722498  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 01:57:22.726128  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:22.732357  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:22.740676  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:57:22.947968  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 01:57:22.967765  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:22.973573  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:22.994326  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:22.995966  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:22.997232  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:23.013574  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:23.064455  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 01:57:23.080483  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:23.104213  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:23.122102  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:23.136435  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:23.160526  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:23.176002  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:23.183243  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:57:23.188354  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 01:57:23.204859  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:23.206030  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:23.216006  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:57:23.374652  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:57:23.375205  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:23.401527  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:23.405658  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:23.408476  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:23.426283  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:23.453933  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:23.465929  6402 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 79 similar messages]
I20240502 01:57:23.489902  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:23.512100  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:57:23.538959  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:57:23.575044  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:23.639930  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 01:57:23.725718  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:23.726076  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:23.730859  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:23.733877  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:23.773181  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:23.773869  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:23.786607  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:23.802127  6135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 01:57:23.821658  5999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 01:57:23.835474  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:57:23.873265  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 01:57:23.928799  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:23.936224  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:23.948460  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:23.962145  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:23.976577  6270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 01:57:23.977609  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:57:23.993640  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 01:57:24.028594  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:24.053635  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:24.098963  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:24.181505  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 01:57:24.189271  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:57:24.190045  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 01:57:24.206806  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:24.241587  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:24.257495  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:24.262804  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:24.316574  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:24.317576  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:24.317618  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:24.333552  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 01:57:24.336221  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:24.343221  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:24.347102  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:24.383797  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 01:57:24.403151  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:24.404693  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:24.461113  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:24.501734  6402 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.133:43379, user_credentials={real_user=slave}} blocked reactor thread for 67828.1us
W20240502 01:57:24.652061  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:24.666903  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:57:24.697633  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 01:57:24.760349  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:24.760509  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:24.783701  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:24.792119  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:24.792371  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:24.792794  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:24.793464  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:24.812250  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:24.827615  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:24.866436  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:24.866981  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:24.905218  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:24.945787  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 01:57:24.970665  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:25.123996  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:25.132896  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:25.147688  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:25.164436  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:25.176169  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:25.181617  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:25.186566  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:25.256705  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:57:25.260695  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:57:25.301559  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:57:25.324039  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:25.442165  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:25.467250  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:25.482445  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:25.484928  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:57:25.494354  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 01:57:25.500682  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:25.539022  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 01:57:25.571280  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:25.589773  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:25.592947  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:25.595146  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:25.640825  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:25.642549  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:25.694324  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 01:57:25.696722  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:25.714103  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:25.720829  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:57:25.737063  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 01:57:25.740468  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:25.762642  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:25.818903 25345 raft_consensus.cc:482] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 01:57:25.819087 25345 raft_consensus.cc:504] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:57:25.819850 25345 leader_election.cc:288] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 01:57:25.821468  6270 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:25.821667  6269 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:25.821787  6269 leader_election.cc:302] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:57:25.822058 25345 raft_consensus.cc:2726] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:57:25.881490  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:57:25.896565  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 01:57:25.913349  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:25.944427  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:25.948282  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:25.979092  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:25.984989  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:25.998736  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:26.154817  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:26.177543  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 01:57:26.195114  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:26.211063  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:26.214015  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:26.222180  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:26.247272  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:26.250627  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:26.251107  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:26.273890  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:57:26.297003  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 01:57:26.468211  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:26.507824  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:26.527284  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:26.527724  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:26.544914  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:57:26.559427  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:57:26.582216  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 01:57:26.616921  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:26.624111  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:26.716581  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:57:26.753278  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 01:57:26.818384  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:26.830405  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:26.839555  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 01:57:26.839658  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:26.852699  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:26.879599  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:26.898674  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:26.917006  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:26.921550  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:26.929847  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 01:57:26.954246  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:26.961833  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:26.974375  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:26.984562  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:27.018716  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:27.019412  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:27.084118  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:57:27.221508  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 01:57:27.250106  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:27.255023  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:27.278491  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:27.281358  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:27.290058  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:27.307384  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:27.322728  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:27.338313  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:57:27.359553  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:27.500761  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:27.517853  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:27.531978  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:27.562525  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:27.574496  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:27.624197  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:27.696610  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:57:27.708657  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:57:27.750244  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:27.801523  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 01:57:27.806895  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:27.822811  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:27.828855  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 01:57:27.840675  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:27.854741  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:27.882779  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:27.893874  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:27.901625  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:57:27.951025  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:27.965061  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:57:27.987437  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:57:27.999102  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 01:57:28.081576  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:28.092634  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:28.115125  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:28.130290  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:28.133103  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:28.149365  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:28.164897  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:28.191742  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:28.195626  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:28.210449  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:28.233929  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:28.274238  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:28.292529  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:28.296707  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:57:28.415539  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:57:28.471215  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:57:28.477667  6402 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 82 similar messages]
I20240502 01:57:28.480978  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:28.487481  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:28.516856  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:28.537261  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:28.545650  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:28.569684  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:28.571357  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:28.617372  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:28.623174  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:57:28.732734  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:57:28.798480  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 01:57:28.832468  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:28.854017  5999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 73 similar messages]
I20240502 01:57:28.866178  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:28.868916  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:28.871587  6136 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240502 01:57:28.871766  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:28.877863  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:28.901808  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:28.922508  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:28.948176  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:28.961593  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:28.975826  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:28.976661  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:29.008183  6270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240502 01:57:29.034092  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:29.037060  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 01:57:29.042048  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:29.059214  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:29.112782  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 01:57:29.236502  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:29.240557  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:29.245126  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:29.255070  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 01:57:29.255720  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 01:57:29.267091  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:29.268918  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:57:29.309789  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 01:57:29.333236  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:29.336903  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:29.360785  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:57:29.383951  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:57:29.390978  6402 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.131:44245, user_credentials={real_user=slave}} blocked reactor thread for 117578us
W20240502 01:57:29.476068  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:29.571079  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:29.579146  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:29.596372  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:29.597723  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:29.634028  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:29.638298  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:29.642382  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 01:57:29.648769  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:29.770799  6515 maintenance_manager.cc:419] P 07cdb213c77c4ffda1f999b66b725399: Scheduling FlushMRSOp(0cecd1b14e474dee825fae040ce1b31e): perf score=0.033342
W20240502 01:57:29.780306  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 01:57:29.841218  6383 maintenance_manager.cc:419] P 94ffd7aaf85b4e17881241b8396fa2f3: Scheduling FlushMRSOp(9a5f486852554e2fa2d07f7b9d686ced): perf score=0.033365
I20240502 01:57:29.843637  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:29.852429  6251 maintenance_manager.cc:419] P a2c90ee440b44b708bccc2e2d5385cc0: Scheduling FlushMRSOp(086509f1ee0a4a81b2f8194b50dbe159): perf score=0.033364
I20240502 01:57:29.871321  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:29.872689  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:29.901568  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:29.934114  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:29.978250  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:30.008356  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:30.112322  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:30.112937  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 01:57:30.155412  6404 maintenance_manager.cc:643] P 07cdb213c77c4ffda1f999b66b725399: FlushMRSOp(0cecd1b14e474dee825fae040ce1b31e) complete. Timing: real 0.384s	user 0.066s	sys 0.018s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":22686,"dirs.queue_time_us":4198,"dirs.run_cpu_time_us":241,"dirs.run_wall_time_us":5941,"drs_written":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4,"lbm_write_time_us":33748,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":37133,"threads_started":3,"wal-append.queue_time_us":9197}
I20240502 01:57:30.163064  6515 maintenance_manager.cc:419] P 07cdb213c77c4ffda1f999b66b725399: Scheduling UndoDeltaBlockGCOp(0cecd1b14e474dee825fae040ce1b31e): 4834 bytes on disk
I20240502 01:57:30.164388  6404 maintenance_manager.cc:643] P 07cdb213c77c4ffda1f999b66b725399: UndoDeltaBlockGCOp(0cecd1b14e474dee825fae040ce1b31e) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":162,"lbm_reads_lt_1ms":4}
W20240502 01:57:30.208073  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:57:30.221657  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:57:30.270522  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 01:57:30.345950  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:30.361256  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:30.365892  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 01:57:30.421641  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:30.432168  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:30.453958  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:30.461118  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:30.471618  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:30.486203  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 01:57:30.493465  6138 maintenance_manager.cc:643] P a2c90ee440b44b708bccc2e2d5385cc0: FlushMRSOp(086509f1ee0a4a81b2f8194b50dbe159) complete. Timing: real 0.638s	user 0.071s	sys 0.010s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":432,"dirs.queue_time_us":18726,"dirs.run_cpu_time_us":282,"dirs.run_wall_time_us":37534,"drs_written":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4,"lbm_write_time_us":956,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":56862,"threads_started":3,"wal-append.queue_time_us":36533}
I20240502 01:57:30.497357 25836 raft_consensus.cc:482] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 01:57:30.497551 25836 raft_consensus.cc:504] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:57:30.498170 25836 leader_election.cc:288] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 01:57:30.498359  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:57:30.501036  6000 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:30.504467  6251 maintenance_manager.cc:419] P a2c90ee440b44b708bccc2e2d5385cc0: Scheduling UndoDeltaBlockGCOp(086509f1ee0a4a81b2f8194b50dbe159): 4921 bytes on disk
I20240502 01:57:30.509217  6138 maintenance_manager.cc:643] P a2c90ee440b44b708bccc2e2d5385cc0: UndoDeltaBlockGCOp(086509f1ee0a4a81b2f8194b50dbe159) complete. Timing: real 0.004s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4}
W20240502 01:57:30.512668  5999 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:30.513072  5999 leader_election.cc:302] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:57:30.515774 26070 raft_consensus.cc:2726] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 01:57:30.543790  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:30.564164  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:30.570827  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:30.610769  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:30.640735  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:30.675678  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 01:57:30.705317  6272 maintenance_manager.cc:643] P 94ffd7aaf85b4e17881241b8396fa2f3: FlushMRSOp(9a5f486852554e2fa2d07f7b9d686ced) complete. Timing: real 0.863s	user 0.065s	sys 0.021s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":47131,"dirs.queue_time_us":24435,"dirs.run_cpu_time_us":269,"dirs.run_wall_time_us":2877,"drs_written":1,"lbm_read_time_us":315,"lbm_reads_lt_1ms":4,"lbm_write_time_us":11031,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":21,"peak_mem_usage":0,"rows_written":635,"thread_start_us":168028,"threads_started":3,"wal-append.queue_time_us":17091}
I20240502 01:57:30.707381  6383 maintenance_manager.cc:419] P 94ffd7aaf85b4e17881241b8396fa2f3: Scheduling UndoDeltaBlockGCOp(9a5f486852554e2fa2d07f7b9d686ced): 4581 bytes on disk
I20240502 01:57:30.736639  6272 maintenance_manager.cc:643] P 94ffd7aaf85b4e17881241b8396fa2f3: UndoDeltaBlockGCOp(9a5f486852554e2fa2d07f7b9d686ced) complete. Timing: real 0.029s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":129,"lbm_reads_lt_1ms":4}
I20240502 01:57:30.738112  6383 maintenance_manager.cc:419] P 94ffd7aaf85b4e17881241b8396fa2f3: Scheduling FlushMRSOp(0cecd1b14e474dee825fae040ce1b31e): perf score=0.033610
I20240502 01:57:30.748370  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:30.767038  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:30.878057  6116 maintenance_manager.cc:419] P 218bc8c1f9ce46059898fc738179adf9: Scheduling FlushMRSOp(89f7cf12d85442749e7d5b259ebca43f): perf score=0.033368
I20240502 01:57:30.928885  6515 maintenance_manager.cc:419] P 07cdb213c77c4ffda1f999b66b725399: Scheduling FlushMRSOp(4eb91f5df851459b8ecf09cbe34fefa5): perf score=0.033375
W20240502 01:57:31.006893  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 01:57:31.017593  6272 maintenance_manager.cc:643] P 94ffd7aaf85b4e17881241b8396fa2f3: FlushMRSOp(0cecd1b14e474dee825fae040ce1b31e) complete. Timing: real 0.279s	user 0.044s	sys 0.014s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":25057,"dirs.run_cpu_time_us":273,"dirs.run_wall_time_us":3965,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1136,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":41711,"threads_started":2,"wal-append.queue_time_us":12263}
I20240502 01:57:31.019008  6383 maintenance_manager.cc:419] P 94ffd7aaf85b4e17881241b8396fa2f3: Scheduling FlushMRSOp(ac3325a06d8841718158e1f5f181a398): perf score=0.033403
W20240502 01:57:31.020022  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:31.026504  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:57:31.034289  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:31.038231  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:57:31.071818  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 01:57:31.130409  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 01:57:31.135563  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:31.171712  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:31.177547  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:31.192515  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:31.200387  6404 maintenance_manager.cc:643] P 07cdb213c77c4ffda1f999b66b725399: FlushMRSOp(4eb91f5df851459b8ecf09cbe34fefa5) complete. Timing: real 0.271s	user 0.044s	sys 0.033s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":18126,"dirs.run_cpu_time_us":263,"dirs.run_wall_time_us":3773,"drs_written":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4345,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":104446,"threads_started":2,"wal-append.queue_time_us":74300}
I20240502 01:57:31.201740  6515 maintenance_manager.cc:419] P 07cdb213c77c4ffda1f999b66b725399: Scheduling UndoDeltaBlockGCOp(4eb91f5df851459b8ecf09cbe34fefa5): 4921 bytes on disk
I20240502 01:57:31.207474  6404 maintenance_manager.cc:643] P 07cdb213c77c4ffda1f999b66b725399: UndoDeltaBlockGCOp(4eb91f5df851459b8ecf09cbe34fefa5) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
I20240502 01:57:31.257833  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:31.259553  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:31.287585  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:57:31.366210  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 01:57:31.373080  6002 maintenance_manager.cc:643] P 218bc8c1f9ce46059898fc738179adf9: FlushMRSOp(89f7cf12d85442749e7d5b259ebca43f) complete. Timing: real 0.494s	user 0.053s	sys 0.032s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":12793,"dirs.queue_time_us":15794,"dirs.run_cpu_time_us":261,"dirs.run_wall_time_us":27268,"drs_written":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1160,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":37034,"threads_started":3,"wal-append.queue_time_us":7992}
I20240502 01:57:31.375306  6116 maintenance_manager.cc:419] P 218bc8c1f9ce46059898fc738179adf9: Scheduling UndoDeltaBlockGCOp(89f7cf12d85442749e7d5b259ebca43f): 4834 bytes on disk
I20240502 01:57:31.380030  6002 maintenance_manager.cc:643] P 218bc8c1f9ce46059898fc738179adf9: UndoDeltaBlockGCOp(89f7cf12d85442749e7d5b259ebca43f) complete. Timing: real 0.002s	user 0.001s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":123,"lbm_reads_lt_1ms":4}
I20240502 01:57:31.421674  6272 maintenance_manager.cc:643] P 94ffd7aaf85b4e17881241b8396fa2f3: FlushMRSOp(ac3325a06d8841718158e1f5f181a398) complete. Timing: real 0.387s	user 0.067s	sys 0.010s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":1532,"dirs.run_cpu_time_us":170,"dirs.run_wall_time_us":2067,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":9540,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":30227,"threads_started":1,"wal-append.queue_time_us":6136}
I20240502 01:57:31.429386  6383 maintenance_manager.cc:419] P 94ffd7aaf85b4e17881241b8396fa2f3: Scheduling UndoDeltaBlockGCOp(0cecd1b14e474dee825fae040ce1b31e): 4834 bytes on disk
I20240502 01:57:31.436560  6272 maintenance_manager.cc:643] P 94ffd7aaf85b4e17881241b8396fa2f3: UndoDeltaBlockGCOp(0cecd1b14e474dee825fae040ce1b31e) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":128,"lbm_reads_lt_1ms":4}
I20240502 01:57:31.438172  6383 maintenance_manager.cc:419] P 94ffd7aaf85b4e17881241b8396fa2f3: Scheduling UndoDeltaBlockGCOp(ac3325a06d8841718158e1f5f181a398): 4581 bytes on disk
I20240502 01:57:31.438961  6272 maintenance_manager.cc:643] P 94ffd7aaf85b4e17881241b8396fa2f3: UndoDeltaBlockGCOp(ac3325a06d8841718158e1f5f181a398) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4}
W20240502 01:57:31.453274  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:31.482281  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:31.595386  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 01:57:31.612221  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:31.617389  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:31.623703  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:57:31.639817  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 01:57:31.664013  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:31.689805  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:31.750036  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:31.769155  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 01:57:31.770290  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:31.804913  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:31.836544  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:31.883684  6116 maintenance_manager.cc:419] P 218bc8c1f9ce46059898fc738179adf9: Scheduling FlushMRSOp(afa5561f8c0e4c0fa9bbbd5da61348c7): perf score=0.033347
I20240502 01:57:31.919813  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:31.922202  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:31.925966  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:57:31.926658  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 01:57:31.938580  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:31.943452  6383 maintenance_manager.cc:419] P 94ffd7aaf85b4e17881241b8396fa2f3: Scheduling FlushMRSOp(afa5561f8c0e4c0fa9bbbd5da61348c7): perf score=0.033369
I20240502 01:57:31.963114  6515 maintenance_manager.cc:419] P 07cdb213c77c4ffda1f999b66b725399: Scheduling FlushMRSOp(02b106ec57b94e44a29947a80a030a23): perf score=0.033374
W20240502 01:57:31.971508  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:31.972820  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 01:57:31.988626  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:32.020002  6251 maintenance_manager.cc:419] P a2c90ee440b44b708bccc2e2d5385cc0: Scheduling FlushMRSOp(ca458907378e42d48adaa0ee647db4cd): perf score=0.033387
W20240502 01:57:32.063897  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:32.119047  6404 maintenance_manager.cc:643] P 07cdb213c77c4ffda1f999b66b725399: FlushMRSOp(02b106ec57b94e44a29947a80a030a23) complete. Timing: real 0.156s	user 0.046s	sys 0.010s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":7444,"dirs.run_cpu_time_us":243,"dirs.run_wall_time_us":13989,"drs_written":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4,"lbm_write_time_us":846,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":14284,"threads_started":2,"wal-append.queue_time_us":6564}
I20240502 01:57:32.120467  6515 maintenance_manager.cc:419] P 07cdb213c77c4ffda1f999b66b725399: Scheduling UndoDeltaBlockGCOp(02b106ec57b94e44a29947a80a030a23): 4919 bytes on disk
I20240502 01:57:32.121160  6404 maintenance_manager.cc:643] P 07cdb213c77c4ffda1f999b66b725399: UndoDeltaBlockGCOp(02b106ec57b94e44a29947a80a030a23) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4}
I20240502 01:57:32.186233  6002 maintenance_manager.cc:643] P 218bc8c1f9ce46059898fc738179adf9: FlushMRSOp(afa5561f8c0e4c0fa9bbbd5da61348c7) complete. Timing: real 0.302s	user 0.038s	sys 0.025s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":14828,"dirs.run_cpu_time_us":270,"dirs.run_wall_time_us":23089,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4145,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":21,"peak_mem_usage":0,"rows_written":635,"thread_start_us":42894,"threads_started":2,"wal-append.queue_time_us":27982}
I20240502 01:57:32.187741  6116 maintenance_manager.cc:419] P 218bc8c1f9ce46059898fc738179adf9: Scheduling UndoDeltaBlockGCOp(afa5561f8c0e4c0fa9bbbd5da61348c7): 4581 bytes on disk
I20240502 01:57:32.188585  6002 maintenance_manager.cc:643] P 218bc8c1f9ce46059898fc738179adf9: UndoDeltaBlockGCOp(afa5561f8c0e4c0fa9bbbd5da61348c7) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":118,"lbm_reads_lt_1ms":4}
I20240502 01:57:32.219822  6272 maintenance_manager.cc:643] P 94ffd7aaf85b4e17881241b8396fa2f3: FlushMRSOp(afa5561f8c0e4c0fa9bbbd5da61348c7) complete. Timing: real 0.276s	user 0.057s	sys 0.023s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":6885,"dirs.run_cpu_time_us":212,"dirs.run_wall_time_us":3756,"drs_written":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4,"lbm_write_time_us":6438,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":20251,"threads_started":2,"wal-append.queue_time_us":13315}
I20240502 01:57:32.221145  6383 maintenance_manager.cc:419] P 94ffd7aaf85b4e17881241b8396fa2f3: Scheduling UndoDeltaBlockGCOp(afa5561f8c0e4c0fa9bbbd5da61348c7): 4581 bytes on disk
I20240502 01:57:32.221838  6272 maintenance_manager.cc:643] P 94ffd7aaf85b4e17881241b8396fa2f3: UndoDeltaBlockGCOp(afa5561f8c0e4c0fa9bbbd5da61348c7) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4}
W20240502 01:57:32.229492  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 01:57:32.232684  6138 maintenance_manager.cc:643] P a2c90ee440b44b708bccc2e2d5385cc0: FlushMRSOp(ca458907378e42d48adaa0ee647db4cd) complete. Timing: real 0.212s	user 0.069s	sys 0.002s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":1947,"dirs.run_cpu_time_us":264,"dirs.run_wall_time_us":4961,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":954,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":30648,"threads_started":2,"wal-append.queue_time_us":20531}
I20240502 01:57:32.234081  6251 maintenance_manager.cc:419] P a2c90ee440b44b708bccc2e2d5385cc0: Scheduling UndoDeltaBlockGCOp(ca458907378e42d48adaa0ee647db4cd): 4834 bytes on disk
I20240502 01:57:32.234766  6138 maintenance_manager.cc:643] P a2c90ee440b44b708bccc2e2d5385cc0: UndoDeltaBlockGCOp(ca458907378e42d48adaa0ee647db4cd) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4}
W20240502 01:57:32.305737  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:57:32.439611  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:32.637662  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:32.673909  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:32.694341  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:57:32.723625  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 01:57:32.780128  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:32.806072  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:32.814210  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 01:57:32.831776  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:32.852053  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 01:57:32.868103  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:32.882077  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:32.898398  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:32.902166  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:32.935367  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:32.961408  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:32.984330  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:33.020012  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:33.021476  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:33.060815  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:57:33.092885  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 01:57:33.127535  6515 maintenance_manager.cc:419] P 07cdb213c77c4ffda1f999b66b725399: Scheduling FlushMRSOp(866f87ea9f7f48edb667b1b2eb3fa239): perf score=0.033349
W20240502 01:57:33.151024  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 01:57:33.193706  6116 maintenance_manager.cc:419] P 218bc8c1f9ce46059898fc738179adf9: Scheduling FlushMRSOp(6984d638991442e3adaed1b24bee95e9): perf score=0.033372
I20240502 01:57:33.196193  6404 maintenance_manager.cc:643] P 07cdb213c77c4ffda1f999b66b725399: FlushMRSOp(866f87ea9f7f48edb667b1b2eb3fa239) complete. Timing: real 0.068s	user 0.027s	sys 0.024s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":6386,"dirs.run_cpu_time_us":253,"dirs.run_wall_time_us":5189,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1059,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":9721,"threads_started":2,"wal-append.queue_time_us":3305}
I20240502 01:57:33.198855  6515 maintenance_manager.cc:419] P 07cdb213c77c4ffda1f999b66b725399: Scheduling FlushMRSOp(6984d638991442e3adaed1b24bee95e9): perf score=0.033364
W20240502 01:57:33.199712  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:33.232543  6383 maintenance_manager.cc:419] P 94ffd7aaf85b4e17881241b8396fa2f3: Scheduling FlushMRSOp(7f2519b0c6404799ad84d120b45a7ea4): perf score=0.033379
I20240502 01:57:33.239197  6251 maintenance_manager.cc:419] P a2c90ee440b44b708bccc2e2d5385cc0: Scheduling FlushMRSOp(7f2519b0c6404799ad84d120b45a7ea4): perf score=0.033392
I20240502 01:57:33.254559  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:33.276655  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:33.297245  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:33.310549  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:33.317262  6404 maintenance_manager.cc:643] P 07cdb213c77c4ffda1f999b66b725399: FlushMRSOp(6984d638991442e3adaed1b24bee95e9) complete. Timing: real 0.118s	user 0.068s	sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":56,"dirs.run_cpu_time_us":163,"dirs.run_wall_time_us":1177,"drs_written":1,"lbm_read_time_us":162,"lbm_reads_lt_1ms":4,"lbm_write_time_us":12271,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":6825,"threads_started":1,"wal-append.queue_time_us":6685}
I20240502 01:57:33.318696  6515 maintenance_manager.cc:419] P 07cdb213c77c4ffda1f999b66b725399: Scheduling UndoDeltaBlockGCOp(866f87ea9f7f48edb667b1b2eb3fa239): 4834 bytes on disk
I20240502 01:57:33.319444  6404 maintenance_manager.cc:643] P 07cdb213c77c4ffda1f999b66b725399: UndoDeltaBlockGCOp(866f87ea9f7f48edb667b1b2eb3fa239) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
W20240502 01:57:33.320722  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 01:57:33.321110  6515 maintenance_manager.cc:419] P 07cdb213c77c4ffda1f999b66b725399: Scheduling UndoDeltaBlockGCOp(6984d638991442e3adaed1b24bee95e9): 4581 bytes on disk
I20240502 01:57:33.321705  6404 maintenance_manager.cc:643] P 07cdb213c77c4ffda1f999b66b725399: UndoDeltaBlockGCOp(6984d638991442e3adaed1b24bee95e9) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
W20240502 01:57:33.324712  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 01:57:33.333022  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:33.357486  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:33.368137  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:33.387322  6272 maintenance_manager.cc:643] P 94ffd7aaf85b4e17881241b8396fa2f3: FlushMRSOp(7f2519b0c6404799ad84d120b45a7ea4) complete. Timing: real 0.154s	user 0.032s	sys 0.017s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":6757,"dirs.run_cpu_time_us":232,"dirs.run_wall_time_us":8354,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":914,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":41280,"threads_started":2,"wal-append.queue_time_us":425}
I20240502 01:57:33.393040  6383 maintenance_manager.cc:419] P 94ffd7aaf85b4e17881241b8396fa2f3: Scheduling UndoDeltaBlockGCOp(7f2519b0c6404799ad84d120b45a7ea4): 4921 bytes on disk
I20240502 01:57:33.393823  6272 maintenance_manager.cc:643] P 94ffd7aaf85b4e17881241b8396fa2f3: UndoDeltaBlockGCOp(7f2519b0c6404799ad84d120b45a7ea4) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
W20240502 01:57:33.412322  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 01:57:33.473775  6138 maintenance_manager.cc:643] P a2c90ee440b44b708bccc2e2d5385cc0: FlushMRSOp(7f2519b0c6404799ad84d120b45a7ea4) complete. Timing: real 0.234s	user 0.022s	sys 0.038s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":22947,"dirs.run_cpu_time_us":261,"dirs.run_wall_time_us":9301,"drs_written":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4,"lbm_write_time_us":832,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":37887,"threads_started":2,"wal-append.queue_time_us":12628}
I20240502 01:57:33.475214  6251 maintenance_manager.cc:419] P a2c90ee440b44b708bccc2e2d5385cc0: Scheduling UndoDeltaBlockGCOp(7f2519b0c6404799ad84d120b45a7ea4): 4921 bytes on disk
I20240502 01:57:33.489894  6002 maintenance_manager.cc:643] P 218bc8c1f9ce46059898fc738179adf9: FlushMRSOp(6984d638991442e3adaed1b24bee95e9) complete. Timing: real 0.296s	user 0.067s	sys 0.049s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":161660,"dirs.run_cpu_time_us":260,"dirs.run_wall_time_us":1278,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":923,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":199867,"threads_started":2,"wal-append.queue_time_us":18561}
I20240502 01:57:33.491220  6116 maintenance_manager.cc:419] P 218bc8c1f9ce46059898fc738179adf9: Scheduling UndoDeltaBlockGCOp(6984d638991442e3adaed1b24bee95e9): 4581 bytes on disk
I20240502 01:57:33.492050  6002 maintenance_manager.cc:643] P 218bc8c1f9ce46059898fc738179adf9: UndoDeltaBlockGCOp(6984d638991442e3adaed1b24bee95e9) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":130,"lbm_reads_lt_1ms":4}
I20240502 01:57:33.498258  6138 maintenance_manager.cc:643] P a2c90ee440b44b708bccc2e2d5385cc0: UndoDeltaBlockGCOp(7f2519b0c6404799ad84d120b45a7ea4) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4}
W20240502 01:57:33.516741  6402 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240502 01:57:33.517678  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:57:33.527185  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:57:33.664840  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:57:33.694075  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 01:57:33.706606  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:33.726547  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:33.734086  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:33.749528  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:33.777117  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:33.789584  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:33.789769  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:33.847532  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:33.861455  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:33.883189  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 01:57:33.905936  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:33.916988  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:33.937397  6000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 01:57:33.962769  6135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240502 01:57:33.993028  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:33.996492  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:34.017036  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:34.076269  6515 maintenance_manager.cc:419] P 07cdb213c77c4ffda1f999b66b725399: Scheduling FlushMRSOp(55de74ae85e541a19008fd2fc2da3ffd): perf score=0.033335
W20240502 01:57:34.086963  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 01:57:34.135290  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 01:57:34.135372  6404 maintenance_manager.cc:643] P 07cdb213c77c4ffda1f999b66b725399: FlushMRSOp(55de74ae85e541a19008fd2fc2da3ffd) complete. Timing: real 0.059s	user 0.047s	sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":5710,"dirs.run_cpu_time_us":253,"dirs.run_wall_time_us":10304,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":751,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":12123,"threads_started":2,"wal-append.queue_time_us":6398}
I20240502 01:57:34.139009  6515 maintenance_manager.cc:419] P 07cdb213c77c4ffda1f999b66b725399: Scheduling UndoDeltaBlockGCOp(55de74ae85e541a19008fd2fc2da3ffd): 4834 bytes on disk
I20240502 01:57:34.139799  6404 maintenance_manager.cc:643] P 07cdb213c77c4ffda1f999b66b725399: UndoDeltaBlockGCOp(55de74ae85e541a19008fd2fc2da3ffd) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":124,"lbm_reads_lt_1ms":4}
I20240502 01:57:34.148166  6383 maintenance_manager.cc:419] P 94ffd7aaf85b4e17881241b8396fa2f3: Scheduling FlushMRSOp(55de74ae85e541a19008fd2fc2da3ffd): perf score=0.033355
I20240502 01:57:34.187055  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:34.190220  6270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240502 01:57:34.192082  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 01:57:34.198074  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:34.210721  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:57:34.212250  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:34.224622  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:34.246191  6116 maintenance_manager.cc:419] P 218bc8c1f9ce46059898fc738179adf9: Scheduling FlushMRSOp(c58d6b2401c047cd9af56a66c2e00105): perf score=0.033371
I20240502 01:57:34.251852  6251 maintenance_manager.cc:419] P a2c90ee440b44b708bccc2e2d5385cc0: Scheduling FlushMRSOp(74eadf4c939c4c6383732cd74c5ec9d6): perf score=0.033382
I20240502 01:57:34.262430  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:34.267926  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:34.268385  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:34.279335  6272 maintenance_manager.cc:643] P 94ffd7aaf85b4e17881241b8396fa2f3: FlushMRSOp(55de74ae85e541a19008fd2fc2da3ffd) complete. Timing: real 0.131s	user 0.038s	sys 0.014s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":851,"dirs.run_cpu_time_us":270,"dirs.run_wall_time_us":2639,"drs_written":1,"lbm_read_time_us":478,"lbm_reads_lt_1ms":4,"lbm_write_time_us":902,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":20535,"threads_started":2,"wal-append.queue_time_us":6576}
I20240502 01:57:34.284359  6383 maintenance_manager.cc:419] P 94ffd7aaf85b4e17881241b8396fa2f3: Scheduling UndoDeltaBlockGCOp(55de74ae85e541a19008fd2fc2da3ffd): 4834 bytes on disk
I20240502 01:57:34.285142  6272 maintenance_manager.cc:643] P 94ffd7aaf85b4e17881241b8396fa2f3: UndoDeltaBlockGCOp(55de74ae85e541a19008fd2fc2da3ffd) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":115,"lbm_reads_lt_1ms":4}
I20240502 01:57:34.286346  6383 maintenance_manager.cc:419] P 94ffd7aaf85b4e17881241b8396fa2f3: Scheduling FlushMRSOp(74eadf4c939c4c6383732cd74c5ec9d6): perf score=0.033386
W20240502 01:57:34.360250  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 01:57:34.437525  6002 maintenance_manager.cc:643] P 218bc8c1f9ce46059898fc738179adf9: FlushMRSOp(c58d6b2401c047cd9af56a66c2e00105) complete. Timing: real 0.191s	user 0.032s	sys 0.014s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":23937,"dirs.run_cpu_time_us":167,"dirs.run_wall_time_us":2366,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":43226,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":25715,"threads_started":2,"wal-append.queue_time_us":968}
I20240502 01:57:34.439642  6272 maintenance_manager.cc:643] P 94ffd7aaf85b4e17881241b8396fa2f3: FlushMRSOp(74eadf4c939c4c6383732cd74c5ec9d6) complete. Timing: real 0.153s	user 0.048s	sys 0.002s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":51,"dirs.run_cpu_time_us":189,"dirs.run_wall_time_us":1703,"drs_written":1,"lbm_read_time_us":1691,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3,"lbm_write_time_us":881,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":4893,"threads_started":1,"wal-append.queue_time_us":4744}
I20240502 01:57:34.441036  6116 maintenance_manager.cc:419] P 218bc8c1f9ce46059898fc738179adf9: Scheduling UndoDeltaBlockGCOp(c58d6b2401c047cd9af56a66c2e00105): 4581 bytes on disk
I20240502 01:57:34.441488  6383 maintenance_manager.cc:419] P 94ffd7aaf85b4e17881241b8396fa2f3: Scheduling UndoDeltaBlockGCOp(74eadf4c939c4c6383732cd74c5ec9d6): 4921 bytes on disk
I20240502 01:57:34.442260  6272 maintenance_manager.cc:643] P 94ffd7aaf85b4e17881241b8396fa2f3: UndoDeltaBlockGCOp(74eadf4c939c4c6383732cd74c5ec9d6) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
I20240502 01:57:34.444829  6002 maintenance_manager.cc:643] P 218bc8c1f9ce46059898fc738179adf9: UndoDeltaBlockGCOp(c58d6b2401c047cd9af56a66c2e00105) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":117,"lbm_reads_lt_1ms":4}
I20240502 01:57:34.461212  6138 maintenance_manager.cc:643] P a2c90ee440b44b708bccc2e2d5385cc0: FlushMRSOp(74eadf4c939c4c6383732cd74c5ec9d6) complete. Timing: real 0.209s	user 0.040s	sys 0.011s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":10296,"dirs.run_cpu_time_us":298,"dirs.run_wall_time_us":9545,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1046,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":16005,"threads_started":2,"wal-append.queue_time_us":460}
I20240502 01:57:34.462764  6251 maintenance_manager.cc:419] P a2c90ee440b44b708bccc2e2d5385cc0: Scheduling UndoDeltaBlockGCOp(74eadf4c939c4c6383732cd74c5ec9d6): 4921 bytes on disk
I20240502 01:57:34.465951  6138 maintenance_manager.cc:643] P a2c90ee440b44b708bccc2e2d5385cc0: UndoDeltaBlockGCOp(74eadf4c939c4c6383732cd74c5ec9d6) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":121,"lbm_reads_lt_1ms":4}
W20240502 01:57:34.469538  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:34.578222  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:34.593600  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:34.613502  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:34.636703  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:34.642993  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:34.674007  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 01:57:34.676393  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:34.713280  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:34.779115  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:57:34.780638  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:57:34.786314  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 01:57:34.855393  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:34.874280  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:34.902189  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:34.902340  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:34.902380  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:34.914373  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:34.997792  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:34.999421  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:35.089908  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 01:57:35.117385  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 01:57:35.174252  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:35.182302  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:35.187345  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:35.219985  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:57:35.219828  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 01:57:35.239590  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:35.240348  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:35.261860  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:35.396327  6515 maintenance_manager.cc:419] P 07cdb213c77c4ffda1f999b66b725399: Scheduling FlushMRSOp(e4d4a1bd80f44b668a2d504e47fb744a): perf score=0.033369
W20240502 01:57:35.397866  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:35.432334  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:35.447414  6383 maintenance_manager.cc:419] P 94ffd7aaf85b4e17881241b8396fa2f3: Scheduling FlushMRSOp(a398da35f18c4df394db82c5bf89aeb1): perf score=0.033382
I20240502 01:57:35.451010  6116 maintenance_manager.cc:419] P 218bc8c1f9ce46059898fc738179adf9: Scheduling FlushMRSOp(a398da35f18c4df394db82c5bf89aeb1): perf score=0.033388
I20240502 01:57:35.460146  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:35.462110  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 01:57:35.470937  6251 maintenance_manager.cc:419] P a2c90ee440b44b708bccc2e2d5385cc0: Scheduling FlushMRSOp(d7922fd756344d14a336ff0c54afdf73): perf score=0.033402
I20240502 01:57:35.497517  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:35.518363  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 01:57:35.524202  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:35.536247  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:35.538152  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:35.562019  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:35.559156  6404 maintenance_manager.cc:643] P 07cdb213c77c4ffda1f999b66b725399: FlushMRSOp(e4d4a1bd80f44b668a2d504e47fb744a) complete. Timing: real 0.163s	user 0.039s	sys 0.015s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":6678,"dirs.run_cpu_time_us":238,"dirs.run_wall_time_us":8267,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":924,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":40604,"threads_started":2,"wal-append.queue_time_us":14491}
I20240502 01:57:35.576386  6515 maintenance_manager.cc:419] P 07cdb213c77c4ffda1f999b66b725399: Scheduling UndoDeltaBlockGCOp(e4d4a1bd80f44b668a2d504e47fb744a): 4921 bytes on disk
I20240502 01:57:35.588544  6404 maintenance_manager.cc:643] P 07cdb213c77c4ffda1f999b66b725399: UndoDeltaBlockGCOp(e4d4a1bd80f44b668a2d504e47fb744a) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":135,"lbm_reads_lt_1ms":4}
I20240502 01:57:35.591081 26644 raft_consensus.cc:482] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 01:57:35.591298 26644 raft_consensus.cc:504] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:57:35.592073 26644 leader_election.cc:288] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 01:57:35.595320  6270 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:35.595672  6269 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:35.595777  6269 leader_election.cc:302] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:57:35.596345 26644 raft_consensus.cc:2726] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 01:57:35.650435  6272 maintenance_manager.cc:643] P 94ffd7aaf85b4e17881241b8396fa2f3: FlushMRSOp(a398da35f18c4df394db82c5bf89aeb1) complete. Timing: real 0.203s	user 0.047s	sys 0.020s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":24422,"dirs.run_cpu_time_us":257,"dirs.run_wall_time_us":2630,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5813,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":59493,"threads_started":2,"wal-append.queue_time_us":34996}
I20240502 01:57:35.652766  6383 maintenance_manager.cc:419] P 94ffd7aaf85b4e17881241b8396fa2f3: Scheduling UndoDeltaBlockGCOp(a398da35f18c4df394db82c5bf89aeb1): 4834 bytes on disk
I20240502 01:57:35.653824  6272 maintenance_manager.cc:643] P 94ffd7aaf85b4e17881241b8396fa2f3: UndoDeltaBlockGCOp(a398da35f18c4df394db82c5bf89aeb1) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
I20240502 01:57:35.695310  6002 maintenance_manager.cc:643] P 218bc8c1f9ce46059898fc738179adf9: FlushMRSOp(a398da35f18c4df394db82c5bf89aeb1) complete. Timing: real 0.244s	user 0.067s	sys 0.028s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":33376,"dirs.run_cpu_time_us":214,"dirs.run_wall_time_us":2789,"drs_written":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4,"lbm_write_time_us":972,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":65352,"threads_started":2,"wal-append.queue_time_us":32180}
I20240502 01:57:35.697751  6116 maintenance_manager.cc:419] P 218bc8c1f9ce46059898fc738179adf9: Scheduling UndoDeltaBlockGCOp(a398da35f18c4df394db82c5bf89aeb1): 4834 bytes on disk
I20240502 01:57:35.705037  6002 maintenance_manager.cc:643] P 218bc8c1f9ce46059898fc738179adf9: UndoDeltaBlockGCOp(a398da35f18c4df394db82c5bf89aeb1) complete. Timing: real 0.007s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":6525,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3}
W20240502 01:57:35.706454  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 01:57:35.715654  6138 maintenance_manager.cc:643] P a2c90ee440b44b708bccc2e2d5385cc0: FlushMRSOp(d7922fd756344d14a336ff0c54afdf73) complete. Timing: real 0.244s	user 0.067s	sys 0.009s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":10538,"dirs.run_cpu_time_us":264,"dirs.run_wall_time_us":4588,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2136,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"spinlock_wait_cycles":6400,"thread_start_us":22471,"threads_started":2,"wal-append.queue_time_us":11777}
I20240502 01:57:35.719188  6251 maintenance_manager.cc:419] P a2c90ee440b44b708bccc2e2d5385cc0: Scheduling UndoDeltaBlockGCOp(d7922fd756344d14a336ff0c54afdf73): 4581 bytes on disk
I20240502 01:57:35.720391  6138 maintenance_manager.cc:643] P a2c90ee440b44b708bccc2e2d5385cc0: UndoDeltaBlockGCOp(d7922fd756344d14a336ff0c54afdf73) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":119,"lbm_reads_lt_1ms":4}
I20240502 01:57:35.722314  6251 maintenance_manager.cc:419] P a2c90ee440b44b708bccc2e2d5385cc0: Scheduling FlushMRSOp(e4d4a1bd80f44b668a2d504e47fb744a): perf score=0.033468
W20240502 01:57:35.726346  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 01:57:35.748330  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 01:57:35.820405  6138 maintenance_manager.cc:643] P a2c90ee440b44b708bccc2e2d5385cc0: FlushMRSOp(e4d4a1bd80f44b668a2d504e47fb744a) complete. Timing: real 0.098s	user 0.045s	sys 0.032s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":50,"dirs.run_cpu_time_us":245,"dirs.run_wall_time_us":2302,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1084,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":36925,"threads_started":1,"wal-append.queue_time_us":35908}
I20240502 01:57:35.821980  6251 maintenance_manager.cc:419] P a2c90ee440b44b708bccc2e2d5385cc0: Scheduling UndoDeltaBlockGCOp(e4d4a1bd80f44b668a2d504e47fb744a): 4921 bytes on disk
I20240502 01:57:35.823101  6138 maintenance_manager.cc:643] P a2c90ee440b44b708bccc2e2d5385cc0: UndoDeltaBlockGCOp(e4d4a1bd80f44b668a2d504e47fb744a) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":452,"lbm_reads_lt_1ms":4}
W20240502 01:57:35.852885  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 01:57:35.870570  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:35.895067  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:35.915099  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:35.984380  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:36.006839  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:36.018914  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 01:57:36.036768  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:36.044550  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:36.055035  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:36.055444  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 01:57:36.125244  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 01:57:36.146660  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:36.176985  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:36.177398  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:36.203660  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:36.221766  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:36.224910  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 01:57:36.264050  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:36.324765  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:36.325389  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:36.336730  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 01:57:36.364567  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:36.380801  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:36.408051  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:36.409888  6383 maintenance_manager.cc:419] P 94ffd7aaf85b4e17881241b8396fa2f3: Scheduling FlushMRSOp(899963a8260a42c4b73fb80eda2ecaa9): perf score=0.033336
I20240502 01:57:36.413276  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:36.460044  6116 maintenance_manager.cc:419] P 218bc8c1f9ce46059898fc738179adf9: Scheduling FlushMRSOp(899963a8260a42c4b73fb80eda2ecaa9): perf score=0.033346
W20240502 01:57:36.514773  6402 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.131:44245, user_credentials={real_user=slave}} blocked reactor thread for 53872.1us
W20240502 01:57:36.522818  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 01:57:36.534776  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:36.558110  6272 maintenance_manager.cc:643] P 94ffd7aaf85b4e17881241b8396fa2f3: FlushMRSOp(899963a8260a42c4b73fb80eda2ecaa9) complete. Timing: real 0.148s	user 0.073s	sys 0.026s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":61784,"dirs.run_cpu_time_us":270,"dirs.run_wall_time_us":5879,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":927,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"spinlock_wait_cycles":6016,"thread_start_us":68455,"threads_started":2,"wal-append.queue_time_us":5560}
I20240502 01:57:36.559618  6383 maintenance_manager.cc:419] P 94ffd7aaf85b4e17881241b8396fa2f3: Scheduling UndoDeltaBlockGCOp(899963a8260a42c4b73fb80eda2ecaa9): 4581 bytes on disk
I20240502 01:57:36.560330  6272 maintenance_manager.cc:643] P 94ffd7aaf85b4e17881241b8396fa2f3: UndoDeltaBlockGCOp(899963a8260a42c4b73fb80eda2ecaa9) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4}
I20240502 01:57:36.577253  6251 maintenance_manager.cc:419] P a2c90ee440b44b708bccc2e2d5385cc0: Scheduling FlushMRSOp(73ed91df7ff04846991c95b30a86ea7e): perf score=0.033375
W20240502 01:57:36.585047  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:36.593394  6515 maintenance_manager.cc:419] P 07cdb213c77c4ffda1f999b66b725399: Scheduling FlushMRSOp(4486b413e5c14ec6953e4895ee5797cf): perf score=0.033377
I20240502 01:57:36.609053  6002 maintenance_manager.cc:643] P 218bc8c1f9ce46059898fc738179adf9: FlushMRSOp(899963a8260a42c4b73fb80eda2ecaa9) complete. Timing: real 0.148s	user 0.070s	sys 0.039s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":10141,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":1312,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":972,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":46644,"threads_started":2,"wal-append.queue_time_us":36377}
I20240502 01:57:36.610414  6116 maintenance_manager.cc:419] P 218bc8c1f9ce46059898fc738179adf9: Scheduling UndoDeltaBlockGCOp(899963a8260a42c4b73fb80eda2ecaa9): 4581 bytes on disk
I20240502 01:57:36.611037  6002 maintenance_manager.cc:643] P 218bc8c1f9ce46059898fc738179adf9: UndoDeltaBlockGCOp(899963a8260a42c4b73fb80eda2ecaa9) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
I20240502 01:57:36.675828  6404 maintenance_manager.cc:643] P 07cdb213c77c4ffda1f999b66b725399: FlushMRSOp(4486b413e5c14ec6953e4895ee5797cf) complete. Timing: real 0.082s	user 0.033s	sys 0.030s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":8985,"dirs.run_cpu_time_us":220,"dirs.run_wall_time_us":1396,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":843,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":15921,"threads_started":2,"wal-append.queue_time_us":6638}
I20240502 01:57:36.677292  6515 maintenance_manager.cc:419] P 07cdb213c77c4ffda1f999b66b725399: Scheduling UndoDeltaBlockGCOp(4486b413e5c14ec6953e4895ee5797cf): 4921 bytes on disk
I20240502 01:57:36.678071  6404 maintenance_manager.cc:643] P 07cdb213c77c4ffda1f999b66b725399: UndoDeltaBlockGCOp(4486b413e5c14ec6953e4895ee5797cf) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":117,"lbm_reads_lt_1ms":4}
I20240502 01:57:36.699427  6138 maintenance_manager.cc:643] P a2c90ee440b44b708bccc2e2d5385cc0: FlushMRSOp(73ed91df7ff04846991c95b30a86ea7e) complete. Timing: real 0.122s	user 0.057s	sys 0.019s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":37759,"dirs.run_cpu_time_us":226,"dirs.run_wall_time_us":1403,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":908,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":38603,"threads_started":2,"wal-append.queue_time_us":1059}
I20240502 01:57:36.700585  6251 maintenance_manager.cc:419] P a2c90ee440b44b708bccc2e2d5385cc0: Scheduling UndoDeltaBlockGCOp(73ed91df7ff04846991c95b30a86ea7e): 4834 bytes on disk
I20240502 01:57:36.701077  6138 maintenance_manager.cc:643] P a2c90ee440b44b708bccc2e2d5385cc0: UndoDeltaBlockGCOp(73ed91df7ff04846991c95b30a86ea7e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
W20240502 01:57:36.708796  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 01:57:36.780001  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:57:36.831609  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 01:57:36.918938  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:36.926555  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 01:57:36.948684  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:36.974898  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:36.986236  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:36.996120  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:37.009112  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:37.043947  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:37.046244  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:37.059845  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:37.066334  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:37.081759  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 01:57:37.104629  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:37.105306  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:37.148411  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:37.165470  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:37.172828  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:37.212612  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:37.224895  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:37.226104  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:37.271209  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 01:57:37.287340  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:37.306766  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:37.323843  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:57:37.330757  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:57:37.470239  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:37.478427  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 01:57:37.553252  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 01:57:37.582875  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 01:57:37.616087  6116 maintenance_manager.cc:419] P 218bc8c1f9ce46059898fc738179adf9: Scheduling FlushMRSOp(4f4088f3523f4cd0afdf0de0f447c7af): perf score=0.033385
I20240502 01:57:37.682922  6515 maintenance_manager.cc:419] P 07cdb213c77c4ffda1f999b66b725399: Scheduling FlushMRSOp(0039dd405cfc4dcea5e7353169e3cd52): perf score=0.033397
W20240502 01:57:37.704556  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 01:57:37.717943  6002 maintenance_manager.cc:643] P 218bc8c1f9ce46059898fc738179adf9: FlushMRSOp(4f4088f3523f4cd0afdf0de0f447c7af) complete. Timing: real 0.102s	user 0.050s	sys 0.028s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":8852,"dirs.run_cpu_time_us":221,"dirs.run_wall_time_us":1475,"drs_written":1,"lbm_read_time_us":125,"lbm_reads_lt_1ms":4,"lbm_write_time_us":876,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"spinlock_wait_cycles":9088,"thread_start_us":31680,"threads_started":2,"wal-append.queue_time_us":23030}
I20240502 01:57:37.719605  6116 maintenance_manager.cc:419] P 218bc8c1f9ce46059898fc738179adf9: Scheduling FlushMRSOp(265a9645e1f141cb88c786b9aef8f90e): perf score=0.033411
W20240502 01:57:37.738251  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:37.826851  6404 maintenance_manager.cc:643] P 07cdb213c77c4ffda1f999b66b725399: FlushMRSOp(0039dd405cfc4dcea5e7353169e3cd52) complete. Timing: real 0.144s	user 0.069s	sys 0.012s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":6430,"dirs.run_cpu_time_us":265,"dirs.run_wall_time_us":5217,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":873,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":9778,"threads_started":2,"wal-append.queue_time_us":460}
I20240502 01:57:37.839964  6515 maintenance_manager.cc:419] P 07cdb213c77c4ffda1f999b66b725399: Scheduling UndoDeltaBlockGCOp(0039dd405cfc4dcea5e7353169e3cd52): 4834 bytes on disk
I20240502 01:57:37.840854  6404 maintenance_manager.cc:643] P 07cdb213c77c4ffda1f999b66b725399: UndoDeltaBlockGCOp(0039dd405cfc4dcea5e7353169e3cd52) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":129,"lbm_reads_lt_1ms":4}
I20240502 01:57:37.849417  6002 maintenance_manager.cc:643] P 218bc8c1f9ce46059898fc738179adf9: FlushMRSOp(265a9645e1f141cb88c786b9aef8f90e) complete. Timing: real 0.124s	user 0.081s	sys 0.003s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":67,"dirs.run_cpu_time_us":221,"dirs.run_wall_time_us":1417,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3009,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":18314,"threads_started":1,"wal-append.queue_time_us":17831}
I20240502 01:57:37.850919  6116 maintenance_manager.cc:419] P 218bc8c1f9ce46059898fc738179adf9: Scheduling UndoDeltaBlockGCOp(4f4088f3523f4cd0afdf0de0f447c7af): 4921 bytes on disk
I20240502 01:57:37.851648  6002 maintenance_manager.cc:643] P 218bc8c1f9ce46059898fc738179adf9: UndoDeltaBlockGCOp(4f4088f3523f4cd0afdf0de0f447c7af) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4}
I20240502 01:57:37.852780  6116 maintenance_manager.cc:419] P 218bc8c1f9ce46059898fc738179adf9: Scheduling UndoDeltaBlockGCOp(265a9645e1f141cb88c786b9aef8f90e): 4581 bytes on disk
I20240502 01:57:37.853364  6002 maintenance_manager.cc:643] P 218bc8c1f9ce46059898fc738179adf9: UndoDeltaBlockGCOp(265a9645e1f141cb88c786b9aef8f90e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
W20240502 01:57:37.950846  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 01:57:37.964511  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:37.993731  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:38.024704  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:38.032092  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:38.070935  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 01:57:38.072861  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:38.094476  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:38.122225  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:38.139423  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:38.148115  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:38.165409  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:38.181309  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:38.191715  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:38.196978  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:38.203111  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 01:57:38.204262  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:38.213522  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 01:57:38.223531  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:38.237655  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:38.246196  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:38.249317  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:38.255568  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:38.276441  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:38.289422  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:57:38.387506  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 01:57:38.389092  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:57:38.450925  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:57:38.524400  6401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 01:57:38.527581  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:38.544875  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:38.545461  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 01:57:38.549193  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:38.551318  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:38.565014  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:38.592420  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:38.597035  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:38.644842  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 01:57:38.708381  6251 maintenance_manager.cc:419] P a2c90ee440b44b708bccc2e2d5385cc0: Scheduling FlushMRSOp(3a7698f6bd4349ae832a22c853d76fea): perf score=0.033343
W20240502 01:57:38.727372  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 01:57:38.781419  6138 maintenance_manager.cc:643] P a2c90ee440b44b708bccc2e2d5385cc0: FlushMRSOp(3a7698f6bd4349ae832a22c853d76fea) complete. Timing: real 0.073s	user 0.030s	sys 0.024s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":9635,"dirs.run_cpu_time_us":289,"dirs.run_wall_time_us":7502,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":942,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":13376,"threads_started":2,"wal-append.queue_time_us":3729}
I20240502 01:57:38.784584  6251 maintenance_manager.cc:419] P a2c90ee440b44b708bccc2e2d5385cc0: Scheduling UndoDeltaBlockGCOp(3a7698f6bd4349ae832a22c853d76fea): 4921 bytes on disk
I20240502 01:57:38.785367  6138 maintenance_manager.cc:643] P a2c90ee440b44b708bccc2e2d5385cc0: UndoDeltaBlockGCOp(3a7698f6bd4349ae832a22c853d76fea) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
W20240502 01:57:38.814046  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:38.834509  6383 maintenance_manager.cc:419] P 94ffd7aaf85b4e17881241b8396fa2f3: Scheduling FlushMRSOp(3a7698f6bd4349ae832a22c853d76fea): perf score=0.033371
I20240502 01:57:38.858049  6116 maintenance_manager.cc:419] P 218bc8c1f9ce46059898fc738179adf9: Scheduling FlushMRSOp(99e74a49b42745888058db67049dc682): perf score=0.033386
I20240502 01:57:38.930747  6272 maintenance_manager.cc:643] P 94ffd7aaf85b4e17881241b8396fa2f3: FlushMRSOp(3a7698f6bd4349ae832a22c853d76fea) complete. Timing: real 0.096s	user 0.042s	sys 0.017s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":6713,"dirs.run_cpu_time_us":255,"dirs.run_wall_time_us":9377,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":829,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":27955,"threads_started":2,"wal-append.queue_time_us":19842}
I20240502 01:57:38.932541  6383 maintenance_manager.cc:419] P 94ffd7aaf85b4e17881241b8396fa2f3: Scheduling FlushMRSOp(d8a6db6ae11e4cb8881aee95bdb0e59d): perf score=0.033396
W20240502 01:57:38.944196  6000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 01:57:38.969966  6136 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240502 01:57:38.989223  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:39.010665  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 01:57:39.015826  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:39.017316  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:39.041709  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:39.049316  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:39.060190  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:39.072769  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:39.088939  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 01:57:39.093119  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:39.104466  6272 maintenance_manager.cc:643] P 94ffd7aaf85b4e17881241b8396fa2f3: FlushMRSOp(d8a6db6ae11e4cb8881aee95bdb0e59d) complete. Timing: real 0.170s	user 0.032s	sys 0.018s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":216,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":1353,"drs_written":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1126,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":13934,"threads_started":1,"wal-append.queue_time_us":12920}
I20240502 01:57:39.108012  6383 maintenance_manager.cc:419] P 94ffd7aaf85b4e17881241b8396fa2f3: Scheduling UndoDeltaBlockGCOp(3a7698f6bd4349ae832a22c853d76fea): 4921 bytes on disk
I20240502 01:57:39.108906  6272 maintenance_manager.cc:643] P 94ffd7aaf85b4e17881241b8396fa2f3: UndoDeltaBlockGCOp(3a7698f6bd4349ae832a22c853d76fea) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":119,"lbm_reads_lt_1ms":4}
I20240502 01:57:39.112167  6383 maintenance_manager.cc:419] P 94ffd7aaf85b4e17881241b8396fa2f3: Scheduling UndoDeltaBlockGCOp(d8a6db6ae11e4cb8881aee95bdb0e59d): 4834 bytes on disk
I20240502 01:57:39.113260  6272 maintenance_manager.cc:643] P 94ffd7aaf85b4e17881241b8396fa2f3: UndoDeltaBlockGCOp(d8a6db6ae11e4cb8881aee95bdb0e59d) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
I20240502 01:57:39.122669  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:39.145388  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:39.146397  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:39.140749  6002 maintenance_manager.cc:643] P 218bc8c1f9ce46059898fc738179adf9: FlushMRSOp(99e74a49b42745888058db67049dc682) complete. Timing: real 0.282s	user 0.045s	sys 0.011s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":16903,"dirs.run_cpu_time_us":290,"dirs.run_wall_time_us":20072,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":939,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":100596,"threads_started":2,"wal-append.queue_time_us":54902}
I20240502 01:57:39.173311  6116 maintenance_manager.cc:419] P 218bc8c1f9ce46059898fc738179adf9: Scheduling UndoDeltaBlockGCOp(99e74a49b42745888058db67049dc682): 4581 bytes on disk
I20240502 01:57:39.174002  6002 maintenance_manager.cc:643] P 218bc8c1f9ce46059898fc738179adf9: UndoDeltaBlockGCOp(99e74a49b42745888058db67049dc682) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4}
I20240502 01:57:39.176018  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:39.203472  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:39.227711  6270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240502 01:57:39.236377  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:39.276011  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:57:39.343511  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 01:57:39.378756  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:39.396266  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:39.396620  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:39.411780  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:39.472237  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:39.489300  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:39.538353  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:39.548530  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:57:39.703922  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:39.716583  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 01:57:39.774911  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 01:57:39.854434  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 01:57:39.860668  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:39.864738  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 01:57:39.877084  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:39.905992  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:39.928769  6116 maintenance_manager.cc:419] P 218bc8c1f9ce46059898fc738179adf9: Scheduling FlushMRSOp(5be45a68cff24a75a03b2fcd4b7851f9): perf score=0.033348
I20240502 01:57:39.957072  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:39.959440  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:39.983994  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:39.985456  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 01:57:39.994354  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:40.003245  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:40.011425  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:40.028316  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:40.041046  6251 maintenance_manager.cc:419] P a2c90ee440b44b708bccc2e2d5385cc0: Scheduling FlushMRSOp(f309b40931dd444091869119eab33bbc): perf score=0.033384
W20240502 01:57:40.064640  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 01:57:40.066640  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:40.069743  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:57:40.075974  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 01:57:40.093592  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:40.107189  6515 maintenance_manager.cc:419] P 07cdb213c77c4ffda1f999b66b725399: Scheduling FlushMRSOp(f309b40931dd444091869119eab33bbc): perf score=0.033402
I20240502 01:57:40.120148  6383 maintenance_manager.cc:419] P 94ffd7aaf85b4e17881241b8396fa2f3: Scheduling FlushMRSOp(2cefdfa8851147999c1425c606f36b1f): perf score=0.033403
I20240502 01:57:40.129184  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:40.178220  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:40.221729  6002 maintenance_manager.cc:643] P 218bc8c1f9ce46059898fc738179adf9: FlushMRSOp(5be45a68cff24a75a03b2fcd4b7851f9) complete. Timing: real 0.293s	user 0.044s	sys 0.014s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":7279,"dirs.run_cpu_time_us":241,"dirs.run_wall_time_us":11795,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":969,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":16971,"threads_started":2,"wal-append.queue_time_us":8088}
I20240502 01:57:40.223162  6116 maintenance_manager.cc:419] P 218bc8c1f9ce46059898fc738179adf9: Scheduling UndoDeltaBlockGCOp(5be45a68cff24a75a03b2fcd4b7851f9): 4833 bytes on disk
I20240502 01:57:40.225600  6002 maintenance_manager.cc:643] P 218bc8c1f9ce46059898fc738179adf9: UndoDeltaBlockGCOp(5be45a68cff24a75a03b2fcd4b7851f9) complete. Timing: real 0.002s	user 0.001s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":166,"lbm_reads_lt_1ms":4}
I20240502 01:57:40.249145  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:40.269192  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:40.294167  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:40.305178  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 01:57:40.345160  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:40.368005  6404 maintenance_manager.cc:643] P 07cdb213c77c4ffda1f999b66b725399: FlushMRSOp(f309b40931dd444091869119eab33bbc) complete. Timing: real 0.260s	user 0.053s	sys 0.037s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":101839,"dirs.run_cpu_time_us":269,"dirs.run_wall_time_us":7788,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":877,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":107304,"threads_started":2,"wal-append.queue_time_us":5671}
I20240502 01:57:40.370412  6515 maintenance_manager.cc:419] P 07cdb213c77c4ffda1f999b66b725399: Scheduling FlushMRSOp(2cefdfa8851147999c1425c606f36b1f): perf score=0.033469
I20240502 01:57:40.397130  6138 maintenance_manager.cc:643] P a2c90ee440b44b708bccc2e2d5385cc0: FlushMRSOp(f309b40931dd444091869119eab33bbc) complete. Timing: real 0.356s	user 0.098s	sys 0.095s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":121892,"dirs.run_cpu_time_us":244,"dirs.run_wall_time_us":6301,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":963,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"spinlock_wait_cycles":1280,"thread_start_us":140237,"threads_started":2,"wal-append.queue_time_us":18564}
I20240502 01:57:40.398945  6251 maintenance_manager.cc:419] P a2c90ee440b44b708bccc2e2d5385cc0: Scheduling UndoDeltaBlockGCOp(f309b40931dd444091869119eab33bbc): 4581 bytes on disk
I20240502 01:57:40.406317  6138 maintenance_manager.cc:643] P a2c90ee440b44b708bccc2e2d5385cc0: UndoDeltaBlockGCOp(f309b40931dd444091869119eab33bbc) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":122,"lbm_reads_lt_1ms":4}
I20240502 01:57:40.426528  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:40.431524  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:40.450132  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:40.463466  6272 maintenance_manager.cc:643] P 94ffd7aaf85b4e17881241b8396fa2f3: FlushMRSOp(2cefdfa8851147999c1425c606f36b1f) complete. Timing: real 0.343s	user 0.036s	sys 0.019s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":30812,"dirs.run_cpu_time_us":250,"dirs.run_wall_time_us":5785,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1125,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":37518,"threads_started":2,"wal-append.queue_time_us":6418}
I20240502 01:57:40.469689  6383 maintenance_manager.cc:419] P 94ffd7aaf85b4e17881241b8396fa2f3: Scheduling UndoDeltaBlockGCOp(2cefdfa8851147999c1425c606f36b1f): 4921 bytes on disk
I20240502 01:57:40.470557  6272 maintenance_manager.cc:643] P 94ffd7aaf85b4e17881241b8396fa2f3: UndoDeltaBlockGCOp(2cefdfa8851147999c1425c606f36b1f) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4}
W20240502 01:57:40.480525  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 01:57:40.484179  6404 maintenance_manager.cc:643] P 07cdb213c77c4ffda1f999b66b725399: FlushMRSOp(2cefdfa8851147999c1425c606f36b1f) complete. Timing: real 0.114s	user 0.040s	sys 0.010s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":49,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":1444,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1409,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":9703,"threads_started":1,"wal-append.queue_time_us":2987}
I20240502 01:57:40.486227  6515 maintenance_manager.cc:419] P 07cdb213c77c4ffda1f999b66b725399: Scheduling UndoDeltaBlockGCOp(2cefdfa8851147999c1425c606f36b1f): 4921 bytes on disk
I20240502 01:57:40.487468  6404 maintenance_manager.cc:643] P 07cdb213c77c4ffda1f999b66b725399: UndoDeltaBlockGCOp(2cefdfa8851147999c1425c606f36b1f) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
I20240502 01:57:40.488688  6515 maintenance_manager.cc:419] P 07cdb213c77c4ffda1f999b66b725399: Scheduling UndoDeltaBlockGCOp(f309b40931dd444091869119eab33bbc): 4581 bytes on disk
I20240502 01:57:40.492627  6404 maintenance_manager.cc:643] P 07cdb213c77c4ffda1f999b66b725399: UndoDeltaBlockGCOp(f309b40931dd444091869119eab33bbc) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
W20240502 01:57:40.533885  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 01:57:40.647164  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 01:57:40.669809  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 01:57:40.676792  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:40.680008  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:40.683784  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:40.685081  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:40.699626  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:40.715185  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:40.720166  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:40.792374  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 01:57:40.840880  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 01:57:40.892534  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:57:40.923779  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:40.977694  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 01:57:40.995101  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:40.996366  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:40.997314  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 01:57:41.014173  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:41.036614  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:41.066510  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:41.076391  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:41.117039  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 01:57:41.128778  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 01:57:41.217993  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:41.224974  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:41.242656  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:41.245363  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:41.255098  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:41.272903  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:41.304185  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:41.438464  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:41.472112  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:41.490815  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:41.495488  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:41.510136  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:41.548559  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:41.562203  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:41.730441  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:41.743072  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:41.746397  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:41.764957  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:41.773514  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:41.782769  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:41.797106  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:41.896284  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:57:41.920617  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 01:57:41.935835  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:57:41.936558  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:41.999315  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:42.027027  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:42.047297  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:42.047533  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:42.056169  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:42.080842  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:42.091413  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 01:57:42.091836  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:42.099838  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 01:57:42.109230  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:42.109848  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:42.154670  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:42.205082  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:42.219975  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:42.317960  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 01:57:42.337221  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 01:57:42.362726  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:42.367404  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:42.420360  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 01:57:42.473906  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:42.485217  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:42.502876  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:42.503459  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 01:57:42.509410  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:42.520875  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:42.535356  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 01:57:42.545698  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:42.548902  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:57:42.559854  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 01:57:42.570247  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:42.587527  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 01:57:42.592026  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:42.609338  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:42.609556  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:42.615514  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:42.659176  6000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.131:44245, user_credentials={real_user=slave}} blocked reactor thread for 110853us
W20240502 01:57:42.711647  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 01:57:42.712021  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:42.781944  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:42.861274  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 01:57:42.917449  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:42.936952  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:42.950933  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:42.956564  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:42.976745  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:42.979959  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:43.017205  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:43.088033  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 01:57:43.133482  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:43.154145  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 01:57:43.157496  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:43.177256  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:43.178499  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:43.203701  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:43.213435  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:43.379732  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:43.427855  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:43.438087  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:57:43.445225  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 01:57:43.446406  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:43.452247  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 01:57:43.454794  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:43.458415  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:43.502893  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:43.513334  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 01:57:43.513641  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:43.522156  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:43.529356  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 01:57:43.539177  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:57:43.546020  6402 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240502 01:57:43.548671  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 01:57:43.750392  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:43.765515  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:43.789892  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:43.809211  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:43.819473  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:43.876891  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:43.884229  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:43.961498  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:43.972086  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:43.998245  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:44.010020  6136 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 01:57:44.011528  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:44.013998  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:44.017146  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:44.042811  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:44.096791  6000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 01:57:44.172803  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 01:57:44.206835  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:44.208866  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:44.211357  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:44.237354  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:44.239849  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:44.244760  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:44.340169  6269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240502 01:57:44.373831  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:44.514843  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 01:57:44.519263  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 01:57:44.522955  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:44.526190  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:44.527108  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:44.529840  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:44.542968  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:44.560133  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:44.582863  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:44.659821  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 01:57:44.660022  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 01:57:44.690577  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:44.700047  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:44.713878  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:44.741277  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 01:57:44.745735  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:44.788705  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:44.800194  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 01:57:44.805330  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 01:57:44.823949  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:44.975111  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:57:44.976480  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:44.978622  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:45.004120  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:45.014829  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:45.019029  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:45.023120  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:45.024889  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:45.042503  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:45.082753  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:57:45.122226  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:57:45.126992  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 01:57:45.200017  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:45.209398  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:45.213582  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:45.216101  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:45.231350  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 01:57:45.247491  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:45.262941  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:45.269301  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:57:45.460021  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:45.460599  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 01:57:45.460767  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:45.480351  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:45.500524  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:45.516595  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:45.572183  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:45.574158  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:45.585945  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:45.658169  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 01:57:45.702632  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:45.725602  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 01:57:45.729992  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:45.730885  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:45.735949  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:45.777659  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:45.809101  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:45.823781  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 01:57:45.826932 27922 raft_consensus.cc:482] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 01:57:45.827128 27922 raft_consensus.cc:504] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:57:45.827839 27922 leader_election.cc:288] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 01:57:45.834240  6270 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:45.848183  6269 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:45.848490  6269 leader_election.cc:302] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:57:45.848886 27922 raft_consensus.cc:2726] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:57:45.852545  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 01:57:45.892125  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 01:57:45.935286  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:45.996978  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:46.000990  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:57:46.032655  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 01:57:46.033311  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:46.048535  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:46.076305  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:46.079629  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:46.099490  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 01:57:46.107061  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:46.127614  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:46.222141  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:57:46.244894  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 01:57:46.269273  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:46.321144  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:46.346062  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:46.367892  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:46.388013  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:46.392390  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:46.437327  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:46.560681  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:46.562004  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:46.563248  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:46.569379  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:46.600348  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:46.621095  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:46.635232  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:46.837278  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:46.929733  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:46.956784  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:46.958415  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:46.965895  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:57:46.982123  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 01:57:46.982908  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:46.990447  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:47.026382  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:47.101549  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:47.110126  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:47.132812  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:47.147192  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 01:57:47.156072  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:47.183403  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 01:57:47.196574  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:47.200884  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 01:57:47.207326  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 01:57:47.225214  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:47.243053  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 01:57:47.283246  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:47.304646  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:47.326629  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:47.326936  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:47.345528  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:47.365105  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:47.406188  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:47.430610  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:47.442454 27922 raft_consensus.cc:482] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 01:57:47.442656 27922 raft_consensus.cc:504] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:57:47.443383 27922 leader_election.cc:288] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 01:57:47.452162  6269 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:47.465992  6270 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:47.466173  6270 leader_election.cc:302] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:57:47.466552 27922 raft_consensus.cc:2726] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:57:47.517206  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:47.559939  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:57:47.578261  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 01:57:47.584972  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:47.616560  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:47.617380  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 01:57:47.631937  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:47.632529  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:47.636166  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:47.681200  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:47.689205  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 01:57:47.720832  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 01:57:47.722272  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:47.768617  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:57:47.854048  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 01:57:47.908553  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:47.938701  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:47.949512  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:47.952384  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:47.964293  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:47.971107  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 01:57:47.985469  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:48.014575  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:48.016419  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:48.023953  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:48.056380  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:48.102195  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:48.104305  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:48.128613  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 01:57:48.134186  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:48.239735  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 01:57:48.295287  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:48.337644 28548 raft_consensus.cc:482] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 01:57:48.337898 28548 raft_consensus.cc:504] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:57:48.338680 28548 leader_election.cc:288] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 01:57:48.342361  5999 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:48.342614  6000 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:48.342707  6000 leader_election.cc:302] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:57:48.343160 28548 raft_consensus.cc:2726] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 01:57:48.354195  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:48.363107  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:48.431881  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 01:57:48.433969  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:48.441944  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:48.450301  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:48.481472  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:57:48.506772  6268 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.135:40543, user_credentials={real_user=slave}} blocked reactor thread for 77697.9us
W20240502 01:57:48.512658  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:57:48.518589  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 01:57:48.557145  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:48.558383  6401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 01:57:48.560406  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:57:48.585026  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 01:57:48.612303  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:48.634567  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:48.638549  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:48.653529  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 01:57:48.660030  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:48.676779  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:48.700327  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:48.738032  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:48.788682  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:48.894223  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:48.944162  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:48.958101  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:48.958101  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:48.981819  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:48.996860  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:48.999264  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:49.113982  6000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 72 similar messages]
I20240502 01:57:49.185881  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:49.195734  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:49.197223  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:49.211967  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:49.255481  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:49.260886  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:49.297513  6136 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 01:57:49.302181  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 01:57:49.302487  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:49.393182  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 01:57:49.404109  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:49.423030  6269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240502 01:57:49.466086  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 01:57:49.546537  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:49.561472  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:49.564373  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:49.592633  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:49.634919  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 01:57:49.649470  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:49.705881  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 01:57:49.706274  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 01:57:49.707130  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:49.787742  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:49.826460  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 01:57:49.833453  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:49.849846  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:49.862650  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 01:57:49.888414  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:49.906000  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:49.921977  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:49.946194  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:49.971740  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:49.973834  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 01:57:49.977582  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:49.986647  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:49.995296  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 01:57:50.011034  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:50.023119  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:50.036688  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:50.090052  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:50.155717  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 01:57:50.243456  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 01:57:50.245056  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:57:50.294732  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 01:57:50.361613  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:50.377106  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:50.379712  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:50.394431  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 01:57:50.415570  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:50.425257  8048 maintenance_manager.cc:419] P 60298ffea1dd4c25a38765c2ec9ec9d5: Scheduling FlushMRSOp(0cecd1b14e474dee825fae040ce1b31e): perf score=0.033357
W20240502 01:57:50.432504  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:50.474179  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:50.486685  8316 maintenance_manager.cc:419] P 436822a397664c54b8d1b9eb3bc6cff4: Scheduling FlushMRSOp(89f7cf12d85442749e7d5b259ebca43f): perf score=0.033366
I20240502 01:57:50.504662  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:50.530895  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:50.545091  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:50.546519  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:50.554922  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:50.590317  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:50.598745  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:50.605811  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:50.624249  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:57:50.628000  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:57:50.676738  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 01:57:50.825259  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 01:57:50.826920  7937 maintenance_manager.cc:643] P 60298ffea1dd4c25a38765c2ec9ec9d5: FlushMRSOp(0cecd1b14e474dee825fae040ce1b31e) complete. Timing: real 0.401s	user 0.084s	sys 0.022s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":37529,"dirs.queue_time_us":12879,"dirs.run_cpu_time_us":362,"dirs.run_wall_time_us":5813,"drs_written":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4,"lbm_write_time_us":941,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":65500,"threads_started":3,"wal-append.queue_time_us":12051}
I20240502 01:57:50.830163  8048 maintenance_manager.cc:419] P 60298ffea1dd4c25a38765c2ec9ec9d5: Scheduling UndoDeltaBlockGCOp(0cecd1b14e474dee825fae040ce1b31e): 4834 bytes on disk
I20240502 01:57:50.831410  7937 maintenance_manager.cc:643] P 60298ffea1dd4c25a38765c2ec9ec9d5: UndoDeltaBlockGCOp(0cecd1b14e474dee825fae040ce1b31e) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":122,"lbm_reads_lt_1ms":4}
I20240502 01:57:50.832579  8048 maintenance_manager.cc:419] P 60298ffea1dd4c25a38765c2ec9ec9d5: Scheduling FlushMRSOp(ac3325a06d8841718158e1f5f181a398): perf score=0.033341
W20240502 01:57:50.843256  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 01:57:50.891342  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 01:57:50.903323  8206 maintenance_manager.cc:643] P 436822a397664c54b8d1b9eb3bc6cff4: FlushMRSOp(89f7cf12d85442749e7d5b259ebca43f) complete. Timing: real 0.416s	user 0.079s	sys 0.013s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":41569,"dirs.queue_time_us":9679,"dirs.run_cpu_time_us":233,"dirs.run_wall_time_us":1017,"drs_written":1,"lbm_read_time_us":128,"lbm_reads_lt_1ms":4,"lbm_write_time_us":8114,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":62505,"threads_started":3,"wal-append.queue_time_us":10859}
I20240502 01:57:50.905680  8316 maintenance_manager.cc:419] P 436822a397664c54b8d1b9eb3bc6cff4: Scheduling FlushMRSOp(e4d4a1bd80f44b668a2d504e47fb744a): perf score=0.033481
W20240502 01:57:50.951143  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:50.976616  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:50.988730  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 01:57:51.016556  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 01:57:51.018067  7937 maintenance_manager.cc:643] P 60298ffea1dd4c25a38765c2ec9ec9d5: FlushMRSOp(ac3325a06d8841718158e1f5f181a398) complete. Timing: real 0.185s	user 0.055s	sys 0.003s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":49,"dirs.run_cpu_time_us":189,"dirs.run_wall_time_us":1381,"drs_written":1,"lbm_read_time_us":838,"lbm_reads_lt_1ms":4,"lbm_write_time_us":901,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":20852,"threads_started":1,"wal-append.queue_time_us":20258}
I20240502 01:57:51.020155  8048 maintenance_manager.cc:419] P 60298ffea1dd4c25a38765c2ec9ec9d5: Scheduling FlushMRSOp(4486b413e5c14ec6953e4895ee5797cf): perf score=0.033358
I20240502 01:57:51.028472  8206 maintenance_manager.cc:643] P 436822a397664c54b8d1b9eb3bc6cff4: FlushMRSOp(e4d4a1bd80f44b668a2d504e47fb744a) complete. Timing: real 0.117s	user 0.042s	sys 0.014s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":60,"dirs.run_cpu_time_us":246,"dirs.run_wall_time_us":1527,"drs_written":1,"lbm_read_time_us":152,"lbm_reads_lt_1ms":4,"lbm_write_time_us":916,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":2575,"threads_started":1,"wal-append.queue_time_us":2446}
I20240502 01:57:51.030982  8316 maintenance_manager.cc:419] P 436822a397664c54b8d1b9eb3bc6cff4: Scheduling UndoDeltaBlockGCOp(e4d4a1bd80f44b668a2d504e47fb744a): 4921 bytes on disk
I20240502 01:57:51.032210  8206 maintenance_manager.cc:643] P 436822a397664c54b8d1b9eb3bc6cff4: UndoDeltaBlockGCOp(e4d4a1bd80f44b668a2d504e47fb744a) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":126,"lbm_reads_lt_1ms":4}
I20240502 01:57:51.033617  8316 maintenance_manager.cc:419] P 436822a397664c54b8d1b9eb3bc6cff4: Scheduling UndoDeltaBlockGCOp(89f7cf12d85442749e7d5b259ebca43f): 4834 bytes on disk
I20240502 01:57:51.034130  8206 maintenance_manager.cc:643] P 436822a397664c54b8d1b9eb3bc6cff4: UndoDeltaBlockGCOp(89f7cf12d85442749e7d5b259ebca43f) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4}
I20240502 01:57:51.047050  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:51.094429  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 01:57:51.097456  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:51.114665  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:51.154425  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:51.154603  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:51.180992  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:51.249537  7937 maintenance_manager.cc:643] P 60298ffea1dd4c25a38765c2ec9ec9d5: FlushMRSOp(4486b413e5c14ec6953e4895ee5797cf) complete. Timing: real 0.229s	user 0.054s	sys 0.003s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":55,"dirs.run_cpu_time_us":191,"dirs.run_wall_time_us":1425,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5202,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":21,"peak_mem_usage":0,"rows_written":689,"thread_start_us":11029,"threads_started":1,"wal-append.queue_time_us":10924}
I20240502 01:57:51.250972  8048 maintenance_manager.cc:419] P 60298ffea1dd4c25a38765c2ec9ec9d5: Scheduling UndoDeltaBlockGCOp(4486b413e5c14ec6953e4895ee5797cf): 4921 bytes on disk
I20240502 01:57:51.256453  7937 maintenance_manager.cc:643] P 60298ffea1dd4c25a38765c2ec9ec9d5: UndoDeltaBlockGCOp(4486b413e5c14ec6953e4895ee5797cf) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":125,"lbm_reads_lt_1ms":4}
I20240502 01:57:51.257941  8048 maintenance_manager.cc:419] P 60298ffea1dd4c25a38765c2ec9ec9d5: Scheduling UndoDeltaBlockGCOp(ac3325a06d8841718158e1f5f181a398): 4581 bytes on disk
I20240502 01:57:51.258736  7937 maintenance_manager.cc:643] P 60298ffea1dd4c25a38765c2ec9ec9d5: UndoDeltaBlockGCOp(ac3325a06d8841718158e1f5f181a398) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":115,"lbm_reads_lt_1ms":4}
W20240502 01:57:51.482497  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:51.533299  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:51.690497  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:51.695914  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:51.759374  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:51.762660  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:51.789031  8316 maintenance_manager.cc:419] P 436822a397664c54b8d1b9eb3bc6cff4: Scheduling FlushMRSOp(99e74a49b42745888058db67049dc682): perf score=0.033351
I20240502 01:57:51.831058  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:51.837126  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:51.894868  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:51.917178  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 01:57:51.938048  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 01:57:51.949117  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:51.952308  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:51.953425  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 01:57:51.966012  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:51.972165  6182 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:51.992548  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 01:57:52.001127  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:52.004344  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:52.120409  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:52.149649  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 01:57:52.155221  8206 maintenance_manager.cc:643] P 436822a397664c54b8d1b9eb3bc6cff4: FlushMRSOp(99e74a49b42745888058db67049dc682) complete. Timing: real 0.366s	user 0.042s	sys 0.007s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":317293,"compiler_manager_pool.run_cpu_time_us":29,"compiler_manager_pool.run_wall_time_us":30,"dirs.queue_time_us":10622,"dirs.run_cpu_time_us":269,"dirs.run_wall_time_us":1544,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1033,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":20280,"threads_started":2,"wal-append.queue_time_us":6925}
I20240502 01:57:52.156956  8316 maintenance_manager.cc:419] P 436822a397664c54b8d1b9eb3bc6cff4: Scheduling UndoDeltaBlockGCOp(99e74a49b42745888058db67049dc682): 4581 bytes on disk
I20240502 01:57:52.157619  8206 maintenance_manager.cc:643] P 436822a397664c54b8d1b9eb3bc6cff4: UndoDeltaBlockGCOp(99e74a49b42745888058db67049dc682) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
W20240502 01:57:52.186551  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 01:57:52.238834  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:52.266551  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:52.288973  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:52.317262  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:52.351567  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:52.353636  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:52.409772  8316 maintenance_manager.cc:419] P 436822a397664c54b8d1b9eb3bc6cff4: Scheduling FlushMRSOp(a398da35f18c4df394db82c5bf89aeb1): perf score=0.033343
W20240502 01:57:52.428094  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 01:57:52.457768  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:52.464625  8206 maintenance_manager.cc:643] P 436822a397664c54b8d1b9eb3bc6cff4: FlushMRSOp(a398da35f18c4df394db82c5bf89aeb1) complete. Timing: real 0.055s	user 0.037s	sys 0.012s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":77,"dirs.run_cpu_time_us":214,"dirs.run_wall_time_us":1665,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":933,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":6222,"threads_started":1,"wal-append.queue_time_us":6300}
I20240502 01:57:52.466594  8316 maintenance_manager.cc:419] P 436822a397664c54b8d1b9eb3bc6cff4: Scheduling UndoDeltaBlockGCOp(a398da35f18c4df394db82c5bf89aeb1): 4834 bytes on disk
I20240502 01:57:52.467262  8206 maintenance_manager.cc:643] P 436822a397664c54b8d1b9eb3bc6cff4: UndoDeltaBlockGCOp(a398da35f18c4df394db82c5bf89aeb1) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
W20240502 01:57:52.473495  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 01:57:52.511157  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 01:57:52.521140  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:52.525449  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:52.536998  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:52.550710  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:52.557351  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 01:57:52.557529  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 01:57:52.561020  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 01:57:52.571244  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:52.586277  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:52.718305  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 01:57:52.727321  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:52.765688  8048 maintenance_manager.cc:419] P 60298ffea1dd4c25a38765c2ec9ec9d5: Scheduling FlushMRSOp(3a7698f6bd4349ae832a22c853d76fea): perf score=0.033396
I20240502 01:57:52.805749  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:52.807976  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:52.828958  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:52.847388  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:52.854647  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:52.855355  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:52.915627  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 01:57:52.925220  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:52.949121  7937 maintenance_manager.cc:643] P 60298ffea1dd4c25a38765c2ec9ec9d5: FlushMRSOp(3a7698f6bd4349ae832a22c853d76fea) complete. Timing: real 0.183s	user 0.066s	sys 0.016s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":6644,"dirs.run_cpu_time_us":222,"dirs.run_wall_time_us":6443,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1079,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":14493,"threads_started":2,"wal-append.queue_time_us":9421}
I20240502 01:57:52.950330  8048 maintenance_manager.cc:419] P 60298ffea1dd4c25a38765c2ec9ec9d5: Scheduling UndoDeltaBlockGCOp(3a7698f6bd4349ae832a22c853d76fea): 4921 bytes on disk
I20240502 01:57:52.950980  7937 maintenance_manager.cc:643] P 60298ffea1dd4c25a38765c2ec9ec9d5: UndoDeltaBlockGCOp(3a7698f6bd4349ae832a22c853d76fea) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
W20240502 01:57:52.971451  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 01:57:53.091060  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:53.096264  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:53.096665  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 01:57:53.102965  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:53.107309  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:53.130837  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:53.147894  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 01:57:53.159821  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:53.176163  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:53.196944  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 01:57:53.227820  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 01:57:53.249966  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:53.281329  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:53.296169  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:53.352653  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:53.357959  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:53.396098  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 01:57:53.402063  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:53.406710  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:53.523036  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 01:57:53.561367  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 01:57:53.612551  6401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 79 similar messages]
I20240502 01:57:53.634086  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:53.645984  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:53.661506  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:53.693650  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:53.717502  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:53.724272  8316 maintenance_manager.cc:419] P 436822a397664c54b8d1b9eb3bc6cff4: Scheduling FlushMRSOp(086509f1ee0a4a81b2f8194b50dbe159): perf score=0.033391
I20240502 01:57:53.729779  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:53.747416  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:53.756016  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:53.773409  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 01:57:53.821014  8206 maintenance_manager.cc:643] P 436822a397664c54b8d1b9eb3bc6cff4: FlushMRSOp(086509f1ee0a4a81b2f8194b50dbe159) complete. Timing: real 0.096s	user 0.063s	sys 0.023s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":6362,"dirs.run_cpu_time_us":253,"dirs.run_wall_time_us":5850,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":866,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":15567,"threads_started":2,"wal-append.queue_time_us":8923}
I20240502 01:57:53.822621  8316 maintenance_manager.cc:419] P 436822a397664c54b8d1b9eb3bc6cff4: Scheduling FlushMRSOp(73ed91df7ff04846991c95b30a86ea7e): perf score=0.033413
I20240502 01:57:53.823418  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:53.825017  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:53.849395  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:53.854439  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 01:57:53.880007  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:53.904661  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:53.922639  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:53.959714  8048 maintenance_manager.cc:419] P 60298ffea1dd4c25a38765c2ec9ec9d5: Scheduling FlushMRSOp(74eadf4c939c4c6383732cd74c5ec9d6): perf score=0.033386
I20240502 01:57:54.009871  8206 maintenance_manager.cc:643] P 436822a397664c54b8d1b9eb3bc6cff4: FlushMRSOp(73ed91df7ff04846991c95b30a86ea7e) complete. Timing: real 0.187s	user 0.065s	sys 0.056s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":57,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":1221,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":968,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"spinlock_wait_cycles":29056,"thread_start_us":88667,"threads_started":1,"wal-append.queue_time_us":88774}
I20240502 01:57:54.011473  8316 maintenance_manager.cc:419] P 436822a397664c54b8d1b9eb3bc6cff4: Scheduling UndoDeltaBlockGCOp(086509f1ee0a4a81b2f8194b50dbe159): 4921 bytes on disk
I20240502 01:57:54.013058  8206 maintenance_manager.cc:643] P 436822a397664c54b8d1b9eb3bc6cff4: UndoDeltaBlockGCOp(086509f1ee0a4a81b2f8194b50dbe159) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4}
I20240502 01:57:54.014329  8316 maintenance_manager.cc:419] P 436822a397664c54b8d1b9eb3bc6cff4: Scheduling UndoDeltaBlockGCOp(73ed91df7ff04846991c95b30a86ea7e): 4834 bytes on disk
I20240502 01:57:54.015264  8206 maintenance_manager.cc:643] P 436822a397664c54b8d1b9eb3bc6cff4: UndoDeltaBlockGCOp(73ed91df7ff04846991c95b30a86ea7e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20240502 01:57:54.031347  8316 maintenance_manager.cc:419] P 436822a397664c54b8d1b9eb3bc6cff4: Scheduling FlushMRSOp(afa5561f8c0e4c0fa9bbbd5da61348c7): perf score=0.033367
I20240502 01:57:54.073959  7937 maintenance_manager.cc:643] P 60298ffea1dd4c25a38765c2ec9ec9d5: FlushMRSOp(74eadf4c939c4c6383732cd74c5ec9d6) complete. Timing: real 0.114s	user 0.065s	sys 0.016s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":11585,"dirs.run_cpu_time_us":235,"dirs.run_wall_time_us":7021,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":774,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":13058,"threads_started":2,"wal-append.queue_time_us":5901}
I20240502 01:57:54.075206  8048 maintenance_manager.cc:419] P 60298ffea1dd4c25a38765c2ec9ec9d5: Scheduling UndoDeltaBlockGCOp(74eadf4c939c4c6383732cd74c5ec9d6): 4921 bytes on disk
I20240502 01:57:54.075959  7937 maintenance_manager.cc:643] P 60298ffea1dd4c25a38765c2ec9ec9d5: UndoDeltaBlockGCOp(74eadf4c939c4c6383732cd74c5ec9d6) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
W20240502 01:57:54.122781  5999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240502 01:57:54.141712  8206 maintenance_manager.cc:643] P 436822a397664c54b8d1b9eb3bc6cff4: FlushMRSOp(afa5561f8c0e4c0fa9bbbd5da61348c7) complete. Timing: real 0.110s	user 0.063s	sys 0.026s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":54,"dirs.run_cpu_time_us":160,"dirs.run_wall_time_us":1362,"drs_written":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4,"lbm_write_time_us":884,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":27023,"threads_started":1,"wal-append.queue_time_us":27132}
I20240502 01:57:54.143414  8316 maintenance_manager.cc:419] P 436822a397664c54b8d1b9eb3bc6cff4: Scheduling UndoDeltaBlockGCOp(afa5561f8c0e4c0fa9bbbd5da61348c7): 4581 bytes on disk
I20240502 01:57:54.144224  8206 maintenance_manager.cc:643] P 436822a397664c54b8d1b9eb3bc6cff4: UndoDeltaBlockGCOp(afa5561f8c0e4c0fa9bbbd5da61348c7) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":187,"lbm_reads_lt_1ms":4}
I20240502 01:57:54.237802  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:54.249894  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:54.252851  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:54.264227  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 01:57:54.272338  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:54.285423  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:54.298617  6136 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 01:57:54.299692  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:54.317379  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:54.327813  8048 maintenance_manager.cc:419] P 60298ffea1dd4c25a38765c2ec9ec9d5: Scheduling FlushMRSOp(6984d638991442e3adaed1b24bee95e9): perf score=0.033367
W20240502 01:57:54.352807  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 01:57:54.353221  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:54.379369  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 01:57:54.449790  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:54.452422  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:54.468068  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:54.520229  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 01:57:54.542901  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 01:57:54.556291  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:54.559726  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:54.565083  7937 maintenance_manager.cc:643] P 60298ffea1dd4c25a38765c2ec9ec9d5: FlushMRSOp(6984d638991442e3adaed1b24bee95e9) complete. Timing: real 0.228s	user 0.087s	sys 0.068s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":66,"dirs.run_cpu_time_us":232,"dirs.run_wall_time_us":1419,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1733,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":117772,"threads_started":1,"wal-append.queue_time_us":117405}
I20240502 01:57:54.566460  8048 maintenance_manager.cc:419] P 60298ffea1dd4c25a38765c2ec9ec9d5: Scheduling UndoDeltaBlockGCOp(6984d638991442e3adaed1b24bee95e9): 4581 bytes on disk
I20240502 01:57:54.567090  7937 maintenance_manager.cc:643] P 60298ffea1dd4c25a38765c2ec9ec9d5: UndoDeltaBlockGCOp(6984d638991442e3adaed1b24bee95e9) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
I20240502 01:57:54.570137  8048 maintenance_manager.cc:419] P 60298ffea1dd4c25a38765c2ec9ec9d5: Scheduling FlushMRSOp(02b106ec57b94e44a29947a80a030a23): perf score=0.033344
I20240502 01:57:54.593297  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:54.654875  6269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 52 similar messages]
I20240502 01:57:54.674197  7937 maintenance_manager.cc:643] P 60298ffea1dd4c25a38765c2ec9ec9d5: FlushMRSOp(02b106ec57b94e44a29947a80a030a23) complete. Timing: real 0.104s	user 0.034s	sys 0.014s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":47,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":1612,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":858,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":2064,"threads_started":1,"wal-append.queue_time_us":2173}
I20240502 01:57:54.675521  8048 maintenance_manager.cc:419] P 60298ffea1dd4c25a38765c2ec9ec9d5: Scheduling UndoDeltaBlockGCOp(02b106ec57b94e44a29947a80a030a23): 4919 bytes on disk
I20240502 01:57:54.676165  7937 maintenance_manager.cc:643] P 60298ffea1dd4c25a38765c2ec9ec9d5: UndoDeltaBlockGCOp(02b106ec57b94e44a29947a80a030a23) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":130,"lbm_reads_lt_1ms":4}
W20240502 01:57:54.712123  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:54.759152  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 01:57:54.792358  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:54.801602  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:54.809659  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:54.827042  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:54.845615  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:54.861389  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:54.913944  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 01:57:54.926580  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 01:57:54.957912  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 01:57:54.963900  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:55.008531  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:55.016052  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:55.044150  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:55.068748  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:55.095530  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:55.110543  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:55.140251  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 01:57:55.150672  8316 maintenance_manager.cc:419] P 436822a397664c54b8d1b9eb3bc6cff4: Scheduling FlushMRSOp(5be45a68cff24a75a03b2fcd4b7851f9): perf score=0.033374
W20240502 01:57:55.172880  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 01:57:55.220062  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 01:57:55.233256  8206 maintenance_manager.cc:643] P 436822a397664c54b8d1b9eb3bc6cff4: FlushMRSOp(5be45a68cff24a75a03b2fcd4b7851f9) complete. Timing: real 0.082s	user 0.030s	sys 0.016s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":389,"dirs.run_cpu_time_us":212,"dirs.run_wall_time_us":2789,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":791,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":3768,"threads_started":2,"wal-append.queue_time_us":3270}
I20240502 01:57:55.234588  8316 maintenance_manager.cc:419] P 436822a397664c54b8d1b9eb3bc6cff4: Scheduling FlushMRSOp(2cefdfa8851147999c1425c606f36b1f): perf score=0.033397
W20240502 01:57:55.251384  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:55.358839  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 01:57:55.372669  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 01:57:55.373364  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:55.376084  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:55.376533  8206 maintenance_manager.cc:643] P 436822a397664c54b8d1b9eb3bc6cff4: FlushMRSOp(2cefdfa8851147999c1425c606f36b1f) complete. Timing: real 0.142s	user 0.082s	sys 0.017s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":57,"dirs.run_cpu_time_us":262,"dirs.run_wall_time_us":1529,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":843,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":49459,"threads_started":1,"wal-append.queue_time_us":48494}
I20240502 01:57:55.382145  8316 maintenance_manager.cc:419] P 436822a397664c54b8d1b9eb3bc6cff4: Scheduling UndoDeltaBlockGCOp(2cefdfa8851147999c1425c606f36b1f): 4921 bytes on disk
W20240502 01:57:55.385457  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:55.393150  8206 maintenance_manager.cc:643] P 436822a397664c54b8d1b9eb3bc6cff4: UndoDeltaBlockGCOp(2cefdfa8851147999c1425c606f36b1f) complete. Timing: real 0.004s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":3872,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3}
I20240502 01:57:55.394587  8316 maintenance_manager.cc:419] P 436822a397664c54b8d1b9eb3bc6cff4: Scheduling UndoDeltaBlockGCOp(5be45a68cff24a75a03b2fcd4b7851f9): 4833 bytes on disk
I20240502 01:57:55.396569  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:55.399075  8206 maintenance_manager.cc:643] P 436822a397664c54b8d1b9eb3bc6cff4: UndoDeltaBlockGCOp(5be45a68cff24a75a03b2fcd4b7851f9) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":118,"lbm_reads_lt_1ms":4}
I20240502 01:57:55.415195  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:55.446692  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:55.453523  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:55.505265  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 01:57:55.543545  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 01:57:55.569797  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 01:57:55.622695  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:55.632288  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 01:57:55.633169  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:55.657339  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:55.671530  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:55.699416  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:55.719205  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:55.742339  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:55.803543  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 01:57:55.816463  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 01:57:55.931896  8048 maintenance_manager.cc:419] P 60298ffea1dd4c25a38765c2ec9ec9d5: Scheduling FlushMRSOp(0039dd405cfc4dcea5e7353169e3cd52): perf score=0.033392
I20240502 01:57:55.934895  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:55.950953  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:55.954098  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:55.958595  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:55.977093  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:55.977967  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:56.093711  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:56.095597  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:56.103474  7937 maintenance_manager.cc:643] P 60298ffea1dd4c25a38765c2ec9ec9d5: FlushMRSOp(0039dd405cfc4dcea5e7353169e3cd52) complete. Timing: real 0.171s	user 0.035s	sys 0.025s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":18850,"dirs.run_cpu_time_us":284,"dirs.run_wall_time_us":7079,"drs_written":1,"lbm_read_time_us":155,"lbm_reads_lt_1ms":4,"lbm_write_time_us":6051,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":21,"peak_mem_usage":0,"rows_written":676,"thread_start_us":18980,"threads_started":2,"wal-append.queue_time_us":504}
I20240502 01:57:56.104959  8048 maintenance_manager.cc:419] P 60298ffea1dd4c25a38765c2ec9ec9d5: Scheduling UndoDeltaBlockGCOp(0039dd405cfc4dcea5e7353169e3cd52): 4834 bytes on disk
I20240502 01:57:56.105682  7937 maintenance_manager.cc:643] P 60298ffea1dd4c25a38765c2ec9ec9d5: UndoDeltaBlockGCOp(0039dd405cfc4dcea5e7353169e3cd52) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4}
W20240502 01:57:56.118551  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 01:57:56.140839  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:56.156668  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:56.169049  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 01:57:56.183812  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 01:57:56.187656  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:56.232545  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:56.253240  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:56.276638  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:56.362411  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 01:57:56.446610  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:56.451526  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:56.480260  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:56.500869  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:57:56.554348  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 01:57:56.562291  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:56.581005  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:56.703054  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:56.708665  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:56.796386  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:56.816247  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:56.841406  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:56.871198  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:56.899500  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 01:57:56.932549  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:56.966058  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:57.003069  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:57.008543  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 01:57:57.015151  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 01:57:57.046056  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 01:57:57.048359  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:57.061591  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:57.062388  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:57.082207  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 01:57:57.131466  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:57.139258  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:57.160130  8316 maintenance_manager.cc:419] P 436822a397664c54b8d1b9eb3bc6cff4: Scheduling FlushMRSOp(866f87ea9f7f48edb667b1b2eb3fa239): perf score=0.033355
I20240502 01:57:57.177909  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:57.265373  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 01:57:57.304100  8206 maintenance_manager.cc:643] P 436822a397664c54b8d1b9eb3bc6cff4: FlushMRSOp(866f87ea9f7f48edb667b1b2eb3fa239) complete. Timing: real 0.144s	user 0.070s	sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":3532,"dirs.run_cpu_time_us":248,"dirs.run_wall_time_us":2402,"drs_written":1,"lbm_read_time_us":141,"lbm_reads_lt_1ms":4,"lbm_write_time_us":883,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":4964,"threads_started":2,"wal-append.queue_time_us":363}
I20240502 01:57:57.305810  8316 maintenance_manager.cc:419] P 436822a397664c54b8d1b9eb3bc6cff4: Scheduling UndoDeltaBlockGCOp(866f87ea9f7f48edb667b1b2eb3fa239): 4834 bytes on disk
I20240502 01:57:57.306802  8206 maintenance_manager.cc:643] P 436822a397664c54b8d1b9eb3bc6cff4: UndoDeltaBlockGCOp(866f87ea9f7f48edb667b1b2eb3fa239) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":117,"lbm_reads_lt_1ms":4}
W20240502 01:57:57.308308  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:57.365556  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 01:57:57.390815  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:57.407294  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:57.411080  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:57.416324  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:57.436774  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:57.444653  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 01:57:57.460573  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:57.469808  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:57.583492  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:57.614159  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 01:57:57.632105  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:57.643812  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:57.668538  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:57.676816  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:57.703441  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:57.742698  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:57.821844  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 01:57:57.829470  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 01:57:57.853404  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 01:57:57.895928  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:57.937458  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 01:57:57.969944  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:57.974505  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:57.992288  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:58.000242  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:58.036617  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:58.039032  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 01:57:58.046659  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 01:57:58.079293  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:58.091796  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 01:57:58.101495  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:58.134280  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:58.152271  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:58.164934  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:58.178324  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 01:57:58.199816  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:58.204684  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:58.238217  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 01:57:58.258263  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:57:58.312932  8316 maintenance_manager.cc:419] P 436822a397664c54b8d1b9eb3bc6cff4: Scheduling FlushMRSOp(55de74ae85e541a19008fd2fc2da3ffd): perf score=0.033340
W20240502 01:57:58.326778  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:58.359690  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 01:57:58.383598  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:58.386808  8206 maintenance_manager.cc:643] P 436822a397664c54b8d1b9eb3bc6cff4: FlushMRSOp(55de74ae85e541a19008fd2fc2da3ffd) complete. Timing: real 0.074s	user 0.032s	sys 0.019s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":6420,"dirs.run_cpu_time_us":318,"dirs.run_wall_time_us":8666,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":884,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"spinlock_wait_cycles":180096,"thread_start_us":6744,"threads_started":2,"wal-append.queue_time_us":335}
I20240502 01:57:58.389664  8316 maintenance_manager.cc:419] P 436822a397664c54b8d1b9eb3bc6cff4: Scheduling UndoDeltaBlockGCOp(55de74ae85e541a19008fd2fc2da3ffd): 4834 bytes on disk
I20240502 01:57:58.392719  8206 maintenance_manager.cc:643] P 436822a397664c54b8d1b9eb3bc6cff4: UndoDeltaBlockGCOp(55de74ae85e541a19008fd2fc2da3ffd) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4}
I20240502 01:57:58.396152  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:58.402362  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:58.407444  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:58.414279  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:58.450969  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:58.492586  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 01:57:58.579306  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 01:57:58.635876  6402 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240502 01:57:58.636835  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 01:57:58.704758  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:58.705511  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:58.706400  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:58.709499  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:58.721518  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:58.738541  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:58.743214  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:58.819175  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 01:57:58.838009  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:57:58.891757  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:58.919586  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:58.929216  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:58.975143  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:57:59.007679  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:59.028170  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:59.128454  6000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240502 01:57:59.129473  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 01:57:59.184244  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:59.214481  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:59.215835  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:57:59.245347  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:59.254796  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 01:57:59.261799  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:59.304345  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:59.310410  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 01:57:59.310524  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:59.325003  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:57:59.377620  6135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240502 01:57:59.470340  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:59.476459  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:59.491755  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:59.500451  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 01:57:59.514007  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:59.518265  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 01:57:59.569823  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:59.588359  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:59.646173  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 01:57:59.676872  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 01:57:59.705765  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:57:59.738898  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:57:59.773520  6270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 52 similar messages]
I20240502 01:57:59.797808  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:57:59.800822  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:57:59.827334  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:57:59.842114  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:57:59.854632  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 01:57:59.871098  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:57:59.876732  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:57:59.925585  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 01:58:00.070933  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:00.083814  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:00.091815  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:00.092281  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:00.126940  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:00.183602  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:00.192867  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:00.214787  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 01:58:00.218238  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 01:58:00.257848  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:00.263980  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 01:58:00.366705  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:00.392531  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:00.398299  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 01:58:00.400038  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:00.420564  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:00.424328  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:00.434180  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:00.441045  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 01:58:00.486831  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 01:58:00.530632  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 01:58:00.570124  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 01:58:00.590072  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 01:58:00.661221  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:00.678882  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:00.700012  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:00.719458  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:00.731498  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:00.793277  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 01:58:00.808393  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:00.818759  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:00.825028  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:00.962373  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 01:58:00.977735  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:00.988143  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:01.024819  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:01.049877  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:01.076417  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:01.084197  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:01.097213  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:01.111550  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 01:58:01.148842  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 01:58:01.208985  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 01:58:01.233031  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:01.298027  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:01.323539  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:01.333765  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 01:58:01.356794  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:01.359077  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:01.382977  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:01.399653  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:01.543622  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:01.563616 30279 raft_consensus.cc:482] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 01:58:01.563825 30279 raft_consensus.cc:504] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:58:01.564594 30279 leader_election.cc:288] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
I20240502 01:58:01.565835  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:01.571805  6270 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:01.572923  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:01.602800  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:01.606410  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:01.611204  6269 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:01.611366  6269 leader_election.cc:302] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:58:01.611655 30279 raft_consensus.cc:2726] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:58:01.624004  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 01:58:01.647732  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:01.652132  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:01.690975  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 01:58:01.716238  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:01.858870  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:01.860762  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:01.862841  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:01.863065  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:01.873338  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:01.900863  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 01:58:01.901499  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:01.939293  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 01:58:01.941624  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 01:58:01.963225  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 01:58:02.050408  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:02.056232  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:02.064663  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:02.078346  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:02.099491  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:02.111501  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:02.125437  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 01:58:02.147902  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:02.168071  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:02.285722  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 01:58:02.333637  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:02.334666  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:02.347718  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:02.351895  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:02.364830  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:02.368468  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:02.529458  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:02.539377  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:02.541937  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:02.563477  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:02.567050  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:02.579102  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 01:58:02.586508  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:02.605743  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:02.626334  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:02.698036  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 01:58:02.723819  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 01:58:02.732650  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 01:58:02.907172  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 01:58:02.930018  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:02.946302  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:02.987718  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:02.988113  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 01:58:02.996471  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:03.014189  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:03.018627  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:03.065311  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:03.076444  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:03.097158  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:03.143682  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:03.173952  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:03.177594  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:03.182363  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:03.184671  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 01:58:03.187968  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:03.192718  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:03.236004  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:03.247152  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:03.251919  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 01:58:03.264573  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:03.282874  6182 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:03.288832  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:03.293076  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 01:58:03.340521  6402 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.133:43379, user_credentials={real_user=slave}} blocked reactor thread for 110953us
W20240502 01:58:03.348934  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 01:58:03.349079  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 01:58:03.403290  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:03.521561  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 01:58:03.566618  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:03.571843  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:03.582523  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:03.589991  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:03.609232  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:03.638669  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:03.642216  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 01:58:03.666731  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 01:58:03.683493  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 01:58:03.684507 30710 raft_consensus.cc:482] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 01:58:03.684680 30710 raft_consensus.cc:504] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:58:03.685410 30710 leader_election.cc:288] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 01:58:03.688023  6269 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:03.688233  6270 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:03.688329  6270 leader_election.cc:302] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:58:03.688825 30710 raft_consensus.cc:2726] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:58:03.732188  6401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 01:58:03.754988  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:03.854682  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:03.861079  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 01:58:03.868410  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 01:58:03.934527  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:03.962019  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:03.967072  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:04.019531  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:04.043550  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:04.044013  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:04.060551  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 01:58:04.061586  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:04.085633  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:04.099524  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:04.132493  6000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240502 01:58:04.136689  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:04.156035  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:04.243332  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:04.246249  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:04.252452  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:04.259121  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 01:58:04.260558  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:04.271517  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 01:58:04.295936  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:04.304596  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:04.337877  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 01:58:04.338773  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:04.342156  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 01:58:04.355676  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 01:58:04.365568  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:04.386951  6135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 01:58:04.457834  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:04.659265  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:04.676815  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:04.679112  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:04.685267  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:04.708334  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:04.712878  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:04.724265  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:04.749581  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:04.766342  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:04.766695  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 01:58:04.817312  6269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 55 similar messages]
I20240502 01:58:04.818791  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:04.828567  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:04.844693  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:04.864672  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:04.923238  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:05.023195  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:05.037632  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:05.042030  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:05.042486  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:05.042927  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:05.094437  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:05.140777  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 01:58:05.248003  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:05.253191  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 01:58:05.297878  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:05.359943  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 01:58:05.374902  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:05.376873  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:05.383989  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 01:58:05.394187  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:05.401053  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:05.420992  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:05.485718  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:05.489974  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 01:58:05.565387  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:05.583050  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 01:58:05.596478  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:05.597568  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:05.610162  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 01:58:05.620594  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:05.647436  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:05.671676  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:05.687301  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 01:58:05.739653  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 01:58:05.763314  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 01:58:05.784309  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:05.856871  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 01:58:05.874452  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 01:58:05.915622  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:05.921053  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:05.924885  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:05.949095  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:05.949887  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:05.951368  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:05.994180  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:05.995266  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:06.005817  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:06.018515  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:06.038774  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 01:58:06.044891  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:06.046119  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:06.048712  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 01:58:06.060068  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:06.089164  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 01:58:06.145044  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 01:58:06.285817  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 01:58:06.394767  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:06.404618  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:06.405157  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:06.440567  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:06.444150  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:06.445531  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:06.469157  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:06.523205  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 01:58:06.531363  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:06.535404  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:06.567679  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:06.577431  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:06.595203  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:06.627987  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:06.651444  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:06.775971  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 01:58:06.789795  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 01:58:06.789989  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 01:58:06.841938  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:06.845630  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 01:58:06.850687  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:06.866662  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:06.875594  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:06.891676  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:06.930773  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:06.961951  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:06.999410  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:07.024860  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:07.032887  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:07.072508  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:07.074891  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:07.116125  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:07.127555  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:07.192788  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 01:58:07.211633  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 01:58:07.291927  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:07.325244  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:07.330243  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:07.339776  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:07.350929  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:07.370865  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:07.405184  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:07.482525  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:07.603032  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 01:58:07.653807  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:07.665920  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:07.686009  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:07.691066  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:07.692124  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:07.715312  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:07.726907  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:07.741746  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:07.782505  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:07.846959  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 01:58:07.853888  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:07.867765  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 01:58:07.907637  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:07.908427  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:07.934631  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:07.937116 29767 raft_consensus.cc:482] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 01:58:07.937311 29767 raft_consensus.cc:504] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:58:07.938045 29767 leader_election.cc:288] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 01:58:07.939931  5999 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:07.941464  6000 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:07.941571  6000 leader_election.cc:302] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:58:07.942284 29767 raft_consensus.cc:2726] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:58:07.997484  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 01:58:08.004170  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:08.032652  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 01:58:08.040174  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 01:58:08.085616  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 01:58:08.099721  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:08.100946  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 01:58:08.105933  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:08.114557  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:08.125646  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:08.129017  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:08.169827  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:08.208088  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 01:58:08.326589  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 01:58:08.339558  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 01:58:08.343273  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 01:58:08.368575  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:08.386622  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 01:58:08.388175  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:08.416304  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:08.421667  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:08.464674  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:08.480990  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:08.483039  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:08.518195  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 01:58:08.526140  6268 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.135:40543, user_credentials={real_user=slave}} blocked reactor thread for 96141.6us
W20240502 01:58:08.561972  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 01:58:08.564231  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:08.588188  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:08.591445  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:08.606190  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:08.629189  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:08.693044  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:08.714498  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:08.757370  6401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 01:58:08.785037  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 01:58:08.842918  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 01:58:08.876045  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:08.896772  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 01:58:08.918041  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:08.939605  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:08.944851  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:08.952719  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:08.976751  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:08.982435  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:09.116994  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:09.139477  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:09.159747  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:09.169270  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 01:58:09.179626  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:09.180462  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:09.196928  5999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240502 01:58:09.201223  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:09.266104  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:09.269742  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 01:58:09.282981  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:09.408859  6135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 57 similar messages]
I20240502 01:58:09.437304  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:09.463505  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:09.465159  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:09.488595  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:09.494586  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 01:58:09.522728  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:09.553964  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:09.638187  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 01:58:09.655965  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 01:58:09.672416  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:09.781618  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:09.818230  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:09.825013  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:09.842773  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:09.849321  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 01:58:09.850090  6270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240502 01:58:09.873303  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:09.880169  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:09.880764  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:09.897208  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:09.922700  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:09.930953  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:09.944296  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:09.962002  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:09.973771  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 01:58:09.988463  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:10.225200  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:10.235774  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:10.236722  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:10.277477  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:10.283339  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:10.314194  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:10.317680  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:10.409534  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 01:58:10.427928  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:10.433744  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 01:58:10.441919  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:10.476456  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:10.486121  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:10.494094  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:10.503213  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 01:58:10.521585  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 01:58:10.555760  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:10.588446  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 01:58:10.597609  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:10.693724  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 01:58:10.710299  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 01:58:10.732355  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 01:58:10.760280  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:10.788834  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:10.791023  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 01:58:10.798509  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:10.862458  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 01:58:10.881322  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:10.889465  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:10.897624  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:10.905153  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:10.909675  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 01:58:10.930270  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 01:58:10.931735  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 01:58:10.932082  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:11.049438  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:11.057492  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 01:58:11.068735  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:11.076990  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 01:58:11.084815  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:11.088342  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:11.093560  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:11.132946  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:11.226497  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 01:58:11.243026  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:11.268081  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:11.283003  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:11.292320  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:11.357165  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:11.389351  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:11.400807  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:11.438282  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:11.517153  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 01:58:11.542663  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 01:58:11.655185  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:11.681041  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 01:58:11.693125  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:11.716001  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:11.730229  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:11.737927  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:11.765225  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:11.766072  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:11.790725  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:11.825856  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:11.841951  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:11.856853  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:11.860337  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:11.891744  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:11.902367  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:12.050136  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 01:58:12.063186  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 01:58:12.091900  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 01:58:12.172914  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:12.187340  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:12.192140  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:12.194869  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:12.208223  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:12.211380  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:12.284399  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:12.331503  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:12.355234  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:12.366299  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:12.371212  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 01:58:12.373632  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:12.404404  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:12.407707  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:12.407954  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 01:58:12.449781  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:12.675444  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:12.681610  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:12.682096  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:12.694152  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:12.739714  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:12.742264  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:12.744354  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 01:58:12.802873  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:12.922955  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 01:58:12.984694  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:12.997893  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:13.009506  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:13.018114  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:13.025741  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 01:58:13.054298  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:13.075781  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 01:58:13.118510  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 01:58:13.126088  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 01:58:13.129439  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:13.144977  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 01:58:13.144991  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:13.145938  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 01:58:13.210489  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 01:58:13.236004  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:13.236372  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 01:58:13.275066  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 01:58:13.332263  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:13.347241  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:13.374610  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:13.426586  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:13.470431  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:13.496259  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:13.500422  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 01:58:13.529922  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 01:58:13.552701  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:13.577237  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 01:58:13.586850  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 01:58:13.673732  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:13.705520  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:13.720961  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:13.728586  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:13.735054  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:13.755578  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 01:58:13.756330  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:13.795205  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:13.797070  6402 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 81 similar messages]
I20240502 01:58:13.890905  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:13.893815  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:13.941592  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:13.941972  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:13.962994  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:13.969513  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:14.010840  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:14.023972  6270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.133:43379, user_credentials={real_user=slave}} blocked reactor thread for 70975.3us
W20240502 01:58:14.130050  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 01:58:14.167186  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:14.168872  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 01:58:14.223913  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:14.234418  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:14.260718  6000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240502 01:58:14.269340  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:14.281024  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:14.290392  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 01:58:14.302673  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:14.372452  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:14.466643  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:14.480615  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:14.489449  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:14.510365  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:14.528416  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:14.560973  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:14.561779  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 01:58:14.642807  6135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 01:58:14.652024  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 01:58:14.661514  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:14.690637  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:14.780933  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:14.799767  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:14.800889  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:14.801668  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:14.830240  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 01:58:14.834918  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:14.846450  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:14.931236  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 01:58:15.009119  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 01:58:15.034581  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:15.041709  6269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240502 01:58:15.094926  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:15.096988  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:15.124579  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:15.156920  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:15.168603  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:15.190716  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:15.238075  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:15.259404  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:15.274070  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:15.284404  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 01:58:15.311895  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 01:58:15.320796  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:15.321249  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:15.342286  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:15.360952  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:15.410786  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 01:58:15.562156  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 01:58:15.565378  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 01:58:15.570753  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 01:58:15.572158  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:15.605216  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:15.637009  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:15.639143  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 01:58:15.645787  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:15.649447  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:15.657040  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:15.663815  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:15.681807  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:15.696544  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:15.707463  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:15.741943  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:15.757401  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 01:58:15.765146  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 01:58:15.766341  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:15.790469  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:15.824409  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 01:58:15.853930  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:15.929932  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 01:58:15.954447  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:15.958099  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 01:58:16.055300  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:16.071185  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:16.074689  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:16.094846  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 01:58:16.100244  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:16.106578  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:16.147150  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:16.203358  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 01:58:16.288640  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:16.291112  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 01:58:16.297966  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:16.312075  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:16.315557  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:16.348984  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:16.413061  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:16.450352  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 01:58:16.461972  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 01:58:16.463052  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:16.478113  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:16.632938  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:16.658277  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:16.661818  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:16.694806  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 01:58:16.697825  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:16.703059  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:16.774921 32731 raft_consensus.cc:482] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 01:58:16.776423 32731 raft_consensus.cc:504] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
W20240502 01:58:16.782094  6269 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:16.781797 32731 leader_election.cc:288] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
I20240502 01:58:16.790071  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:16.801121  6270 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:16.801415  6270 leader_election.cc:302] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:58:16.801795 32637 raft_consensus.cc:2726] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:58:16.810717  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:16.879907  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:16.911979  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 01:58:16.975663  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:16.980160  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:16.982314  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:17.026461  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:17.077301  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:17.081691  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:17.116031  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 01:58:17.186935  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 01:58:17.257472  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 01:58:17.257898  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:17.265764  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:17.294093  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:17.297237  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:17.330124  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:17.357164  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:17.361481  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 01:58:17.381136  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:17.387554  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:17.524571  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 01:58:17.553886  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:17.564070  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:17.569964  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:17.591852  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 01:58:17.592059  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:17.600217  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:17.663619  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:17.673316  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:17.676403  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:17.692687  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 01:58:17.728739  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:17.749205  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:17.757416  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:17.766050  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:17.853052  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:17.943784  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 01:58:17.980842  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:17.988173  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 01:58:17.994138  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:17.996085  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:18.001385  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:18.001744  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:18.004757  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:18.011884  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:18.085992  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 01:58:18.138010  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 01:58:18.227756  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 01:58:18.235564  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 01:58:18.239238  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 01:58:18.276485  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:18.283388  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 01:58:18.293934  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:18.313509  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:18.324374  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:18.359725  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:18.381119  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:18.414654  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:18.428033  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:18.474630  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:18.498947  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:18.523481  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 01:58:18.557617  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:18.566408  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:18.611191  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 01:58:18.627259  6402 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.133:43379, user_credentials={real_user=slave}} blocked reactor thread for 138615us
W20240502 01:58:18.639724  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 01:58:18.639102  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 01:58:18.664418  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:18.681665  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:18.738505  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 01:58:18.823571  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 01:58:18.849843  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:18.869964  6401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 01:58:18.873932  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:18.904996  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:18.917135  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:18.942204  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:18.942327  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:18.943034  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:18.948966  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:18.961336  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 01:58:19.068028  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 01:58:19.141479  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:19.184896  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:19.185107  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:19.200105  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:19.230842  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 01:58:19.239418  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:19.279240  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:19.304044  6000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240502 01:58:19.328152  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:19.338616  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:19.340755  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:19.367697  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:19.406725  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:19.409142  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:19.459993  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:19.490535  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:19.502193  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 01:58:19.577279  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 01:58:19.652248  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:19.656344  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:19.680869  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:19.681519  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:19.691215  6135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240502 01:58:19.711565  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:19.729637  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:19.799554  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:19.822196  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:19.826332  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:19.841231  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:19.859695  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:19.894047  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 01:58:19.902060  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:19.908629  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 01:58:19.937707  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:19.975466  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 01:58:20.107580  6269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 01:58:20.112957  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:20.127893  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:20.128341  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:20.131479  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:20.145407  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:20.186405  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 01:58:20.202778  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:20.204604  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 01:58:20.231444  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:20.247223  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 01:58:20.337277  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:20.362080  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:20.368147  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:20.385776  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:20.401666  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:20.422844  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 01:58:20.474442  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:20.478847  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:20.524658  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 01:58:20.596599  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 01:58:20.602136  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 01:58:20.641263  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:20.650822  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:20.671737  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 01:58:20.725160  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:20.735311  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:20.764780  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 01:58:20.804019  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 01:58:20.851178  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:20.851565  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:20.858587  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:20.890471  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:21.026366  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 01:58:21.106920  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:21.115047  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:21.135337  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:21.137253  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:21.144078  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 01:58:21.149331  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 01:58:21.152710  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:21.157323  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:21.168224  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:21.172050  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:21.193555  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:21.235163  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:21.257122  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:21.302603  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 01:58:21.313310  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:21.318416  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:21.341928  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 01:58:21.376276  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 01:58:21.425246  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:21.505972  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:21.514919  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:21.523530  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:21.535063  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:21.535954  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:21.584216  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:21.611445  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 01:58:21.633433  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:21.756142  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 01:58:21.885515  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:21.895426  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 01:58:21.897017  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:21.898963  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:21.909593  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:21.981122  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:22.005959  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:22.011492  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:22.011929  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:22.012948  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:22.030488  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 01:58:22.047456  6402 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.133:43379, user_credentials={real_user=slave}} blocked reactor thread for 45850.4us
I20240502 01:58:22.056110  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:22.084724  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:22.084761  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:22.103107  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:22.130374  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 01:58:22.256899  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:22.353188  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:22.354189  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:22.361824  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:22.389947  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:22.415030  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:22.456938  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 01:58:22.461977  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:22.542728  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 01:58:22.564533  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:22.597887  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:22.605470  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 01:58:22.608796  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:22.615768  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:22.647133  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 01:58:22.667168  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:22.685977  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:22.694365  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:22.714622  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:22.842033  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 01:58:22.883831  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:22.888319  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:22.921916  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 01:58:22.944877  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:22.947850  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 01:58:22.962888  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:22.973302  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:23.000540  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:23.001837  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:23.016142  1154 raft_consensus.cc:482] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 01:58:23.016367  1154 raft_consensus.cc:504] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:58:23.017171  1154 leader_election.cc:288] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 01:58:23.020671  6269 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:23.021040  6270 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:23.021137  6270 leader_election.cc:302] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:58:23.021838  1154 raft_consensus.cc:2726] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:58:23.069162  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 01:58:23.134824  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 01:58:23.203066  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:23.252941  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 01:58:23.261623  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:23.270474  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:23.278501  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:23.278690  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 01:58:23.292486  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 01:58:23.298234  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:23.304941  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:23.332312  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:23.348279  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 01:58:23.369372  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:23.394405  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 01:58:23.412982  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:23.422857  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:23.484055  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:23.484191  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:23.492321  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:23.555560  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:23.631373  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:23.708068  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 01:58:23.753834  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:23.753897  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 01:58:23.781675  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:23.799736  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:23.833001  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:23.846688  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:23.849978  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:23.890259  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:23.898782  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 01:58:23.958547  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 01:58:23.979826  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 01:58:23.990644  6402 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 78 similar messages]
I20240502 01:58:23.993487  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:24.004202  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:24.018524  1232 raft_consensus.cc:482] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 01:58:24.018744  1232 raft_consensus.cc:504] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:58:24.019538  1232 leader_election.cc:288] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
I20240502 01:58:24.029760  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:24.048538  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 01:58:24.054172  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:24.057019  5999 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:24.057322  6000 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:24.057446  6000 leader_election.cc:302] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:58:24.057971  1232 raft_consensus.cc:2726] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 01:58:24.076992  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:24.098551  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 01:58:24.107678  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:24.170121  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:24.226918  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 01:58:24.271427  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:24.271576  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:24.320046  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:24.345274  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:24.364653  6000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240502 01:58:24.379645  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:24.409610  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:24.460181  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:24.575985  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:24.584900  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 01:58:24.597563  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:24.630381  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:24.631753  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:24.643518  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:24.672363  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:24.683197  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:24.724809  6135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240502 01:58:24.775315  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:24.796620  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 01:58:24.805514  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:24.811030  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:24.847376  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:24.848753  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:24.858875  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:24.886159  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:25.040763  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:25.055840  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:25.056862  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:25.059571  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:25.061815  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:25.071309  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:25.101307  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:25.191419  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 01:58:25.201876  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 01:58:25.213354  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 01:58:25.250947  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 01:58:25.276232  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:25.278410  6269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240502 01:58:25.331651  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 01:58:25.345732  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:25.354862  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:25.365187  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 01:58:25.368597  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:25.377908  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:25.418124  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 01:58:25.418262  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:25.474654  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:25.538245  8316 maintenance_manager.cc:419] P 436822a397664c54b8d1b9eb3bc6cff4: Scheduling FlushMRSOp(ac3325a06d8841718158e1f5f181a398): perf score=0.033398
W20240502 01:58:25.544669  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 01:58:25.603891  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 01:58:25.618247  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:25.632506  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:25.685428  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:25.729883  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 01:58:25.759500  6268 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.135:40543, user_credentials={real_user=slave}} blocked reactor thread for 70277.4us
I20240502 01:58:25.786586  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:25.790472  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:25.792536  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:25.795055  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:25.798354  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:25.835513  5998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.135:40543, user_credentials={real_user=slave}} blocked reactor thread for 54125.1us
W20240502 01:58:25.841677  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 01:58:25.847622  8206 maintenance_manager.cc:643] P 436822a397664c54b8d1b9eb3bc6cff4: FlushMRSOp(ac3325a06d8841718158e1f5f181a398) complete. Timing: real 0.309s	user 0.135s	sys 0.106s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":90666,"dirs.run_cpu_time_us":214,"dirs.run_wall_time_us":2254,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":895,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":243267,"threads_started":2,"wal-append.queue_time_us":152836}
I20240502 01:58:25.849331  8316 maintenance_manager.cc:419] P 436822a397664c54b8d1b9eb3bc6cff4: Scheduling UndoDeltaBlockGCOp(ac3325a06d8841718158e1f5f181a398): 4581 bytes on disk
I20240502 01:58:25.849998  8206 maintenance_manager.cc:643] P 436822a397664c54b8d1b9eb3bc6cff4: UndoDeltaBlockGCOp(ac3325a06d8841718158e1f5f181a398) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4}
W20240502 01:58:25.918128  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 01:58:25.986614  8048 maintenance_manager.cc:419] P 60298ffea1dd4c25a38765c2ec9ec9d5: Scheduling FlushMRSOp(89f7cf12d85442749e7d5b259ebca43f): perf score=0.033394
W20240502 01:58:25.998378  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 01:58:26.025575  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:26.046555  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:26.048278  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:26.051609  7937 maintenance_manager.cc:643] P 60298ffea1dd4c25a38765c2ec9ec9d5: FlushMRSOp(89f7cf12d85442749e7d5b259ebca43f) complete. Timing: real 0.065s	user 0.049s	sys 0.009s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":6742,"dirs.run_cpu_time_us":194,"dirs.run_wall_time_us":905,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1013,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":15766,"threads_started":2,"wal-append.queue_time_us":9264}
I20240502 01:58:26.056305  8048 maintenance_manager.cc:419] P 60298ffea1dd4c25a38765c2ec9ec9d5: Scheduling UndoDeltaBlockGCOp(89f7cf12d85442749e7d5b259ebca43f): 4834 bytes on disk
I20240502 01:58:26.057016  7937 maintenance_manager.cc:643] P 60298ffea1dd4c25a38765c2ec9ec9d5: UndoDeltaBlockGCOp(89f7cf12d85442749e7d5b259ebca43f) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4}
I20240502 01:58:26.082237  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:26.082749  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:26.112928  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:26.117560  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 01:58:26.147950  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:26.169636  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 01:58:26.193886  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:26.212769  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:26.231417  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:26.241870  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:26.246464  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:26.274448  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:26.302428  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:26.420280  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 01:58:26.469218  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 01:58:26.500320  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:26.508812  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:26.522724  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:26.524860  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:26.530227  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:26.567953  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:26.581835  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 01:58:26.612476  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 01:58:26.624641  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:26.628873  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 01:58:26.680565  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:26.703590  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 01:58:26.720548  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:26.732306  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:26.764842  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:26.765146  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:26.786511  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:26.789484  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:26.793442  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 01:58:26.891114  1725 raft_consensus.cc:482] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 01:58:26.891367  1725 raft_consensus.cc:504] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:58:26.892261  1725 leader_election.cc:288] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 01:58:26.895396  6269 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:26.913458  6270 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:26.913771  6270 leader_election.cc:302] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:58:26.914171  1725 raft_consensus.cc:2726] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 01:58:27.028141  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:27.033380  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:27.048907  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:27.071195  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:27.154891  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:27.155334  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:27.165920  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:27.182909  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 01:58:27.253798  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:27.330093  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:27.332039  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 01:58:27.350160  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:27.350643  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:27.358367  8316 maintenance_manager.cc:419] P 436822a397664c54b8d1b9eb3bc6cff4: Scheduling FlushMRSOp(0039dd405cfc4dcea5e7353169e3cd52): perf score=0.033388
I20240502 01:58:27.373857  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:27.391446  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:27.402474  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:27.415197  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:27.459959  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:27.460654  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:27.491135  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 01:58:27.506680  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:27.533977  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:27.536664  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:27.586437  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:27.616236  8206 maintenance_manager.cc:643] P 436822a397664c54b8d1b9eb3bc6cff4: FlushMRSOp(0039dd405cfc4dcea5e7353169e3cd52) complete. Timing: real 0.258s	user 0.045s	sys 0.024s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":18543,"dirs.run_cpu_time_us":244,"dirs.run_wall_time_us":11796,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":916,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":86506,"threads_started":2,"wal-append.queue_time_us":64867}
I20240502 01:58:27.619611  8316 maintenance_manager.cc:419] P 436822a397664c54b8d1b9eb3bc6cff4: Scheduling UndoDeltaBlockGCOp(0039dd405cfc4dcea5e7353169e3cd52): 4834 bytes on disk
I20240502 01:58:27.620541  8206 maintenance_manager.cc:643] P 436822a397664c54b8d1b9eb3bc6cff4: UndoDeltaBlockGCOp(0039dd405cfc4dcea5e7353169e3cd52) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
W20240502 01:58:27.627251  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 01:58:27.740559  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 01:58:27.790660  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 01:58:27.829790  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:27.846349  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:27.849124  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:27.862864  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:27.879830  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:27.904624  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:27.924458  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 01:58:27.927409  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:27.945235  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 01:58:27.950990  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 01:58:28.038539  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 01:58:28.049947  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 01:58:28.092284  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:28.093117  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:28.093452  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:28.102201  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:28.102898  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:28.106179  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:28.112527  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 01:58:28.166410  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:28.297910  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:28.314286  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:28.321919  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:28.335235  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:28.337339  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:28.369560  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:28.374482  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:28.412194  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 01:58:28.437017  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 01:58:28.505913  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 01:58:28.523182  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 01:58:28.554405  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:28.605230  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:28.619901  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:28.626463  8316 maintenance_manager.cc:419] P 436822a397664c54b8d1b9eb3bc6cff4: Scheduling FlushMRSOp(9a5f486852554e2fa2d07f7b9d686ced): perf score=0.033375
I20240502 01:58:28.646979  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:28.663555  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:28.685070  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:28.708922  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:28.716101  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:28.806345  8206 maintenance_manager.cc:643] P 436822a397664c54b8d1b9eb3bc6cff4: FlushMRSOp(9a5f486852554e2fa2d07f7b9d686ced) complete. Timing: real 0.180s	user 0.080s	sys 0.019s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":13530,"dirs.run_cpu_time_us":275,"dirs.run_wall_time_us":1558,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":903,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":49212,"threads_started":2,"wal-append.queue_time_us":42439}
I20240502 01:58:28.810189  8316 maintenance_manager.cc:419] P 436822a397664c54b8d1b9eb3bc6cff4: Scheduling UndoDeltaBlockGCOp(9a5f486852554e2fa2d07f7b9d686ced): 4581 bytes on disk
I20240502 01:58:28.812855  8206 maintenance_manager.cc:643] P 436822a397664c54b8d1b9eb3bc6cff4: UndoDeltaBlockGCOp(9a5f486852554e2fa2d07f7b9d686ced) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
W20240502 01:58:28.845085  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 01:58:28.867226  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 01:58:28.869441  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:28.869657  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:28.900381  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:28.920536  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:28.923537  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:28.936113  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:28.953841  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 01:58:28.968200  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 01:58:28.992537  6402 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240502 01:58:29.001477  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:29.023989  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 01:58:29.022795  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 01:58:29.160013  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 01:58:29.178493  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:29.183478  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:29.191422  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:29.204380  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:29.210474  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:29.213254  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:29.246822  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:29.384166  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 01:58:29.398500  6000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240502 01:58:29.404496  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:29.405972  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:29.406451  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:29.420979  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:29.429170  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:29.430145  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:29.472386  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:29.559556  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:29.582883  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:29.600309  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:29.621359  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:29.622525  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:29.636669  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:29.699307  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:29.725750  6135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 01:58:29.727278  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 01:58:29.847566  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 01:58:29.878659  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 01:58:29.888279  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:29.910840  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 01:58:29.944989  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:29.962298  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:29.966859  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:29.972762  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:29.997473  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:30.002977  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:30.032644  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:30.065665  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:30.075071  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:30.098055  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:30.111907  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:30.153368  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:30.256088  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 01:58:30.287637  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:30.292833  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 01:58:30.307044  6269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240502 01:58:30.312050  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 01:58:30.348600  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 01:58:30.401042  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 01:58:30.401154  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 01:58:30.416517  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:30.461871  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:30.461843  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:30.465204  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:30.509608  6402 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.133:43379, user_credentials={real_user=slave}} blocked reactor thread for 48721.4us
I20240502 01:58:30.516739  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:30.520078  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 01:58:30.533469  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:30.560551  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:30.561012  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:30.562726  6271 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.130:38197, user_credentials={real_user=slave}} blocked reactor thread for 76486.9us
I20240502 01:58:30.565011  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:30.600334  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:30.621776  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 01:58:30.629500  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:30.655634  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:30.678617  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:30.735411  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 01:58:30.760859  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:30.777783  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 01:58:30.924158  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:30.976047  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 01:58:30.978706  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:30.979650  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:30.985214  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:30.986294  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:31.017040  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:31.021526  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:31.068310  6402 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.133:43379, user_credentials={real_user=slave}} blocked reactor thread for 50897.6us
W20240502 01:58:31.104835  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 01:58:31.129952  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:31.134968  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:31.136066  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:31.142796  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:31.164144  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:31.182551  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:31.272480  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:31.273821  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 01:58:31.316149  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:31.329530  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 01:58:31.401679  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 01:58:31.416724  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 01:58:31.424533  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 01:58:31.463120  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:31.465658  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:31.479027  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:31.494369  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:31.497871  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:31.515432  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:31.588892  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 01:58:31.669157  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:31.784292  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 01:58:31.790086  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:31.819976  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:31.859642  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:31.864236  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:31.899387  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:31.932982  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 01:58:31.944986  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:31.953565  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:32.054438  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:32.055769  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:32.084015  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:32.084641  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:32.088495  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:32.114473  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:32.144841  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:32.267786  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 01:58:32.295186  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 01:58:32.337747  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:32.362390  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:32.374536  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:32.388984  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:32.397784  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:32.409644  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:32.425022  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 01:58:32.429625  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:32.443332  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:32.443832  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:32.447538  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:32.467408  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:32.495340  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 01:58:32.508296  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 01:58:32.546548  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:32.560128  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:32.680594  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:32.808286  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:32.815038  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:32.826275  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:32.826484  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:32.850008  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:32.859786  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 01:58:32.871100  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 01:58:32.878880  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:32.910248  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:32.932518  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 01:58:33.010035  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:33.028718  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 01:58:33.032598  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 01:58:33.069854  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:33.070204  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:33.092725  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:33.098098  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:33.106631  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:33.106981  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:33.166039  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 01:58:33.220669  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 01:58:33.246834  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 01:58:33.285310  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:33.287972  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:33.292171  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 01:58:33.312101  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:33.320142  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:33.324286  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:33.336823  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 01:58:33.341339  8048 maintenance_manager.cc:419] P 60298ffea1dd4c25a38765c2ec9ec9d5: Scheduling FlushMRSOp(866f87ea9f7f48edb667b1b2eb3fa239): perf score=0.033376
I20240502 01:58:33.344354  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:33.392124  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:33.467751  7937 maintenance_manager.cc:643] P 60298ffea1dd4c25a38765c2ec9ec9d5: FlushMRSOp(866f87ea9f7f48edb667b1b2eb3fa239) complete. Timing: real 0.126s	user 0.049s	sys 0.001s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":13317,"dirs.run_cpu_time_us":230,"dirs.run_wall_time_us":11846,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":894,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":24674,"threads_started":2,"wal-append.queue_time_us":11355}
I20240502 01:58:33.469208  8048 maintenance_manager.cc:419] P 60298ffea1dd4c25a38765c2ec9ec9d5: Scheduling UndoDeltaBlockGCOp(866f87ea9f7f48edb667b1b2eb3fa239): 4834 bytes on disk
I20240502 01:58:33.479843  7937 maintenance_manager.cc:643] P 60298ffea1dd4c25a38765c2ec9ec9d5: UndoDeltaBlockGCOp(866f87ea9f7f48edb667b1b2eb3fa239) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":124,"lbm_reads_lt_1ms":4}
W20240502 01:58:33.521631  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 01:58:33.523747  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:33.569715  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:33.576180  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 01:58:33.609750  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:33.659003  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 01:58:33.676941  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:33.679617  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:33.685194  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:33.743309  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 01:58:33.803602  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:33.807555  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:33.809592  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:33.819133  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:33.825294  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:33.854961  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:33.873889  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:33.913431  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 01:58:33.942341  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:34.000245  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 01:58:34.008605  6402 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 01:58:34.119433  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:34.189527  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 01:58:34.208959  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 01:58:34.211568  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:34.249470  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:34.250315  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:34.251185  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:34.273707  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:34.308912  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:34.354887  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:34.369027  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:34.373600  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:34.374148  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 01:58:34.402324  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:34.446292  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:34.451505  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:34.452425  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:34.473404  6000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 01:58:34.502557  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 01:58:34.644644  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:34.645952  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:34.668643  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:34.679288  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:34.706615  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:34.742635  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:34.818675  6135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 01:58:34.836211  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:34.836213  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 01:58:34.847075  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:34.951742  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 01:58:34.961656  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 01:58:34.995956  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:35.000720  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:35.021127  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:35.041296  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:35.053092  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:35.053297  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:35.076324  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 01:58:35.093765  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:35.159977  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 01:58:35.247124  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:35.254674  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:35.268692  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 01:58:35.272805  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:35.274298  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:35.297427  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:35.323552  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:35.326257  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:35.356819  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 01:58:35.437255  6269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 01:58:35.451812  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 01:58:35.543145  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:35.559486  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:35.593514  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:35.604979  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 01:58:35.629452  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 01:58:35.649036  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:35.656176  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 01:58:35.657641  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 01:58:35.659962  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:35.662004  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:35.662922  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:35.794505  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 01:58:35.857723  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:35.866412  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:35.891352  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:35.902552  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:35.923352  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:35.924425  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:35.968770  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:36.007918  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 01:58:36.068852  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 01:58:36.079851  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:36.136587  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 01:58:36.155170  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:36.199678  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:36.216306  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:36.231635  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 01:58:36.270958  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:36.298753  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:36.317776  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 01:58:36.325455  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:36.407171  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:36.504420  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:36.510777  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:36.530210  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:36.542304  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:36.547935  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:36.550249  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:36.564018  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 01:58:36.586668  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:36.588218  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 01:58:36.650697  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 01:58:36.691306  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:36.703064  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:36.705449  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:36.707619  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:36.730106  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:36.743804  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:36.792630  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 01:58:36.873497  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 01:58:36.886723  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:36.926633  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:37.025406  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:37.026255  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:37.026949  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:37.027216  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:37.063356  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:37.110520  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:37.133253  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 01:58:37.240753  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 01:58:37.261302  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:37.265239  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:37.280025  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:37.310943  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:37.313752  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:37.322043  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:37.418956  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 01:58:37.420044  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 01:58:37.449025  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:37.480571  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 01:58:37.489910  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:37.496464  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:37.515202  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:37.525998  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:37.574476  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:37.582118  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 01:58:37.632145  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:37.643719  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:37.695564  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 01:58:37.720821  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 01:58:37.835147  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:37.861685  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:37.871338  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:37.882089  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:37.902470  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:37.915387  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:37.934046  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:37.945542  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:37.950443  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:37.967084  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:37.976521  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:38.001964  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:38.027747  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:38.051107  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:38.073836  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 01:58:38.105157  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 01:58:38.127533  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 01:58:38.244728  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:38.253819  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:38.256897  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:38.257207  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 01:58:38.270294  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:38.273254  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:38.288412  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:38.297804  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 01:58:38.327452  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 01:58:38.469914  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 01:58:38.508860  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:38.513815  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:38.520251  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:38.560204  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:38.560638  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:38.579072  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:38.580523  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:38.594172  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:38.602416  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 01:58:38.619148  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:38.657258  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 01:58:38.699185  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:38.704833  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 01:58:38.728096  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:38.751680  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:38.753254  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:38.769346  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 01:58:38.797775  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:38.950438  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:38.971509  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 01:58:38.989079  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 01:58:38.999747  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:39.028846  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:39.036954  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:39.044591  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:39.065302  6402 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 84 similar messages]
W20240502 01:58:39.069968  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 01:58:39.073872  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:39.090056  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:39.130203  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 01:58:39.150157  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:39.295501  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:39.301995  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 01:58:39.305397  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:39.316646  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:39.324223  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:39.341476  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:39.345943  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:39.369143  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:39.417627  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:39.458355  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 01:58:39.504721  6000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240502 01:58:39.540150  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:39.546614  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:39.588393  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:39.619437  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 01:58:39.667641  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:39.673913  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:39.704999  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:39.734900  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 01:58:39.828286  6135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 01:58:39.836022  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:39.884711  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 01:58:39.939539  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:39.949035  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:39.968796  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:39.972107  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:39.976748  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:39.983083  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:39.996491  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 01:58:39.997577  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:40.017130  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 01:58:40.054571  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 01:58:40.231477  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:40.239468  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:40.262120  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:40.282691  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:40.315522  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:40.390771  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:40.417109  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:40.468037  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 01:58:40.476801  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:40.478098  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 01:58:40.482686  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:40.505081  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:40.512328  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:40.532953  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:40.537093  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 01:58:40.566843  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:40.640368  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:40.670449  6269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 01:58:40.684464  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 01:58:40.700610  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:40.700912  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:40.731348  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:40.740710  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:40.765580  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:40.774035  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 01:58:40.788435  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:40.901738  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 01:58:40.913384  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:41.024947  6134 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.135:40543, user_credentials={real_user=slave}} blocked reactor thread for 46850.4us
W20240502 01:58:41.046521  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 01:58:41.057562  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 01:58:41.078444  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:41.086189  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:41.119843  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 01:58:41.120687  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:41.125248  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:41.129669  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:41.186983  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:41.200855  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 01:58:41.247025  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:41.257800  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 01:58:41.343925  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:41.345047  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:41.366568  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:41.427394  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 01:58:41.429078  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:41.439736  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:41.446259  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 01:58:41.447180  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:41.504113  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:41.525173  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:41.630112  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:41.645437  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:41.649099  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 01:58:41.649314  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:41.671927  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 01:58:41.675727  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:41.700146  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:41.734866  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:41.763213  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 01:58:41.767236  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:41.825472  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 01:58:41.923780  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:41.939435  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:41.955824  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:41.956007  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:41.985806  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:42.049995  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:42.051913  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:42.071128  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 01:58:42.158510  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:42.165246  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:42.181795  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 01:58:42.187371  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:42.198844  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:42.250084  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:42.262653  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 01:58:42.277110  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:42.368778  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:42.375263  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:42.412634  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:42.417733  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:42.433171  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 01:58:42.440208  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:42.440843  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 01:58:42.446187  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:42.457080  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:42.514926  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:42.589452  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 01:58:42.665369  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:42.670701  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 01:58:42.674212  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:42.700477  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:42.736550  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:42.776880  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:42.813505  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:42.832150  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 01:58:42.851200  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:42.963618  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:42.986173  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:43.029309  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 01:58:43.032964  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:43.065039  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:43.081215  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:43.098863  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:43.107167  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:43.115020  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 01:58:43.246261  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 01:58:43.247587  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 01:58:43.252938  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:43.270226  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:43.291334  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 01:58:43.292083  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:43.295908  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:43.300565  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 01:58:43.315575  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:43.323642  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:43.326710  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 01:58:43.347582  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:43.496215  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 01:58:43.566164  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:43.584842  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:43.588199  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 01:58:43.599152  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:43.600253  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:43.621138  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:43.638480  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:43.680290  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:43.708604  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:43.744127  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 01:58:43.834676  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 01:58:43.847338  4144 raft_consensus.cc:482] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 01:58:43.847549  4144 raft_consensus.cc:504] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:58:43.848372  4144 leader_election.cc:288] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
I20240502 01:58:43.848862  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:43.851686  5999 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:43.867663  6000 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:43.868011  6000 leader_election.cc:302] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:58:43.868384  4144 raft_consensus.cc:2726] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 01:58:43.904702  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:43.932808  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:43.954042  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:43.954929  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:43.984124  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:43.998901  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 01:58:44.143123  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:44.151628  6401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 01:58:44.153136  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:44.160230  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:44.173131  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:44.178261  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 01:58:44.183509  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 01:58:44.228801  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:44.229475  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:44.259847  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:44.285113  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:44.374370  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 01:58:44.418336  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:44.421661  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:44.449199  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:44.463836  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 01:58:44.466085  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:44.540578  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 01:58:44.541327  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:44.560729  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:44.565766  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 01:58:44.577095  5999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 01:58:44.626139  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:44.633642  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:44.728291  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:44.754328  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:44.755759  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:44.755764  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:44.762894  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:44.782855  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:44.825909  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 01:58:44.888659  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 01:58:44.906069  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 01:58:44.969872  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:44.983047  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:45.014917  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:45.022452  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:45.031935  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:45.033407  6135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 01:58:45.036856  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 01:58:45.063046  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:45.069819  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:45.216997  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:45.226919  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:45.232347  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:45.243096  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:45.248667  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:45.287467  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:45.342765  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:45.468482  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:45.489574  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:45.492403  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:45.496435  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 01:58:45.496572  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:45.511024  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 01:58:45.548909  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:45.562512  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:45.569020  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 01:58:45.610160  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 01:58:45.619007  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 01:58:45.649721  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:45.658623  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:45.693710  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 01:58:45.751338  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 01:58:45.786891  6270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 01:58:45.788081  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 01:58:45.800206  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:45.826735  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:45.847177  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:45.849413  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:45.906615  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:45.962204  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 01:58:45.962903  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:46.070209  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:46.087496  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 01:58:46.092706  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:46.106456  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:46.110683  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:46.113544  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:46.117652  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:46.142589  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:46.153082  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 01:58:46.153630  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 01:58:46.160557  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:46.217869  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:46.251528  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:46.287824  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:46.292559  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 01:58:46.306962  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:46.329282  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:46.371639  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:46.437629  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 01:58:46.570717  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 01:58:46.577798  4252 raft_consensus.cc:482] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 01:58:46.577993  4252 raft_consensus.cc:504] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:58:46.578754  4252 leader_election.cc:288] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 01:58:46.580762  6270 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:46.584889  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:46.591898  6269 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:46.592105  6269 leader_election.cc:302] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:58:46.592690  4252 raft_consensus.cc:2726] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:58:46.684118  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:46.693939  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 01:58:46.718711  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:46.742740  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:46.780645  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:46.782594  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:46.815143  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:46.819955  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:46.871299  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:46.902289  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:46.911576  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 01:58:46.920567  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:46.922523  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:46.946862  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:46.973618  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:47.036509  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:47.064399  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 01:58:47.106664  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 01:58:47.111459  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:47.139652  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:47.145732  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:47.162364  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:47.166272  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 01:58:47.171260  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:47.199460  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:47.210418  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:47.344182  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 01:58:47.435191  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 01:58:47.442566  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:47.443472  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:47.450755  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 01:58:47.453580  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:47.470209  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:47.479228  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:47.516127  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:47.619278  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:47.623905  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:47.635625  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:47.636029  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 01:58:47.648734  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:47.665513  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:47.692102  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:47.713840  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 01:58:47.715332  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:47.720065  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:47.987191  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:47.993703  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:48.013947  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:48.028525  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 01:58:48.032689  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:48.047730  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:48.049219  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:48.068368  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:48.075114  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:48.093778  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 01:58:48.105762  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 01:58:48.133345  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 01:58:48.177124  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 01:58:48.207911  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:48.222244  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:48.236002  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:48.239475  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 01:58:48.240733  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:48.253330  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 01:58:48.263247  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:48.291823  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 01:58:48.291976  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:48.431167  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 01:58:48.448668  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:48.469280  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:48.479748  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:48.484843  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:48.515028  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:48.537623  6402 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.133:43379, user_credentials={real_user=slave}} blocked reactor thread for 48351.5us
I20240502 01:58:48.550590  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:48.615015  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:48.652827  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:48.727137  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 01:58:48.735667  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:48.753345  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:48.782063  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:48.784915  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:48.794754  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:48.797646  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:48.869369  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 01:58:48.996999  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:49.022881  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:49.039753  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 01:58:49.046545  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:49.052841  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:49.053324  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:49.084414  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 01:58:49.092693  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:49.101414  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:49.168790  6402 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 01:58:49.172034  6268 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.135:40543, user_credentials={real_user=slave}} blocked reactor thread for 46953.8us
W20240502 01:58:49.179497  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 01:58:49.218219  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:49.277181  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 01:58:49.310528  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:49.321370  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:49.332072  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 01:58:49.345058  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:49.372524  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:49.378118  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 01:58:49.387202  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:49.409709  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:49.513147  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:49.520483  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 01:58:49.546154  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 01:58:49.572784  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:49.603703  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:49.650959  5999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 66 similar messages]
I20240502 01:58:49.654616  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:49.659147  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:49.675957  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:49.688418  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 01:58:49.691733  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:49.722425  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:49.821983  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:49.831676  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:49.863144  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:49.886193  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:49.890327  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 01:58:49.903432  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:49.931418  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 01:58:49.939941  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:50.046766  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 01:58:50.072134  6135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 01:58:50.112088  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:50.205806  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:50.208612  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:50.243589  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:50.258530  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:50.260143  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:50.285934  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 01:58:50.337427  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:50.347672  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:50.364606  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:50.390300  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:50.399609  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:50.447070  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:50.457335  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 01:58:50.478436  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:50.483044  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 01:58:50.484230  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:50.503659  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 01:58:50.507140  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 01:58:50.551368  5998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.135:40543, user_credentials={real_user=slave}} blocked reactor thread for 46344.9us
W20240502 01:58:50.551465  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:50.623414  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 01:58:50.670480  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 01:58:50.761026  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:50.812793  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:50.823593  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:50.825286  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:50.842200  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:50.856096  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:50.856793  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:50.886083  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 01:58:50.898171  6270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240502 01:58:50.909284  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:50.924674  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 01:58:50.925247  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 01:58:51.001720  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:51.016443  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:51.023417  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:51.047488  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:51.064733  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:51.108453  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 01:58:51.128136  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:51.233106  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:51.238579  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:51.244417  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 01:58:51.251114  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:51.271416  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:51.297260  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:51.357093  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:51.401881  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:51.504460  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:51.518600  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 01:58:51.583047  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:51.611766  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:51.646749  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:51.669569  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:51.671236  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:51.699798  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:51.711050  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 01:58:51.717651  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:51.734880  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:51.773394  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:51.823845  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 01:58:51.833304  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 01:58:51.842700  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:51.881855  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:51.904753  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:51.908749  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 01:58:51.920431  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 01:58:51.945344  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:51.995499  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:52.073252  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:52.078119  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 01:58:52.084863  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:52.088554  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:52.089097  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:52.104272  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:52.104427  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:52.317663  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 01:58:52.401286  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:52.420886  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:52.421697  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:52.421918  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:52.432459  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 01:58:52.444257  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 01:58:52.471026  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:52.475325  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:52.482910  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:52.500106  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:52.510653  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:52.521948  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:52.554396  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:52.559010  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:52.577180  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:52.611954  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:52.712363  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 01:58:52.714267  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 01:58:52.725319  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 01:58:52.812664  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 01:58:52.859730  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:52.875677  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:52.875996  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:52.908030  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:52.929435  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:52.952203  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:52.970219  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 01:58:52.985056  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 01:58:53.012692  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:53.037019  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:53.051996  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:53.056866  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:53.077312  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 01:58:53.092527  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:53.120242  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:53.129895  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:53.135537  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:53.200803  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 01:58:53.202816  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 01:58:53.319357  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:53.319595  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:53.324499  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:53.341488  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:53.370415  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:53.397994  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:53.404264  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 01:58:53.512605  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 01:58:53.533018  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:53.564494  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:53.580965  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 01:58:53.585158  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 01:58:53.619074  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 01:58:53.645408  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:53.668790  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:53.670914  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:53.697723  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:53.703370  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:53.724015  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:53.759430  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:53.803311  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:53.806735  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:53.854902  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:53.919199  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:53.936975  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:54.105957  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:54.124656  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:54.134835  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:54.144572  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:54.148653  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:54.149744  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 01:58:54.175536  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:54.177080  6401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 01:58:54.179066  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 01:58:54.186847  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:54.189998  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:54.272087  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 01:58:54.292093  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:54.307381  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:54.334827  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:54.337354  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:54.360391  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:54.385571  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 01:58:54.385737  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 01:58:54.389883  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:54.517928  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 01:58:54.590018  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:54.607726  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:54.624819  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:54.684218  6000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 68 similar messages]
I20240502 01:58:54.707350  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:54.712250  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:54.718969  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:54.734319  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 01:58:54.742383  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:54.765565  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:54.978888  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:54.986989  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 01:58:54.996261  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 01:58:54.998678  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:55.011435  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 01:58:55.031040  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:55.032858  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:55.034155  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:55.071393  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:55.126876  6135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240502 01:58:55.136386  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:55.144874  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:55.166172  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:55.168066  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:55.168238  6268 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.135:40543, user_credentials={real_user=slave}} blocked reactor thread for 134318us
I20240502 01:58:55.188912  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:55.206284  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 01:58:55.239750  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:55.247992  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 01:58:55.248608  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 01:58:55.315410  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:55.316484  5750 raft_consensus.cc:482] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 01:58:55.316668  5750 raft_consensus.cc:504] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:58:55.317447  5750 leader_election.cc:288] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 01:58:55.320089  6000 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:55.329264  5999 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:55.329584  5999 leader_election.cc:302] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:58:55.331319  5750 raft_consensus.cc:2726] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:58:55.344666  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 01:58:55.381561  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:55.469417  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 01:58:55.524492  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:55.544402  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:55.559458  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:55.578528  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:55.583081  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:55.588181  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 01:58:55.610162  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:55.611068  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 01:58:55.723683  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:55.726183  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 01:58:55.791127  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 01:58:55.830152  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 01:58:55.847612  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:55.858850  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 01:58:55.870038  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:55.871356  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:55.897830  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:55.902344  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:55.903687  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:55.949911  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:55.976084  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 01:58:56.008853  6270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 01:58:56.042006  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 01:58:56.146266  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:56.147085  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:56.162420  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 01:58:56.178625  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:56.189404  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:56.197921  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:56.207456  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:56.216504  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:56.400586  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:56.409308  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:56.437557  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:56.449272  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:56.463277  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:56.465432  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:56.465984  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:56.491879  6271 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.130:38197, user_credentials={real_user=slave}} blocked reactor thread for 66308.8us
I20240502 01:58:56.655702  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:56.683666  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 01:58:56.696135  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:56.714180  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 01:58:56.722489  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:56.723378  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:56.726943  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:56.743414  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:56.751842  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:56.755795  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 01:58:56.875514  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 01:58:56.879525  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 01:58:56.884701  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 01:58:56.917606  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:56.966344  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:56.973845  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:57.008916  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:57.029601  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:57.063108  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:57.095794  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:57.249660  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 01:58:57.283589  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:58:57.291637  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 01:58:57.313890  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:57.314738  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:57.315347  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:57.331059  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:57.358007  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:57.363461  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:57.365440  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:57.412847  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:57.413815  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:57.425339  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:57.436614  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:57.437636  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:57.478307  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:57.478977  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 01:58:57.565918  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 01:58:57.644706  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 01:58:57.655009  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 01:58:57.669663  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 01:58:57.767201  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:57.770195  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:57.796519  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:57.804108  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:57.852080  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:57.869897  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:57.897197  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 01:58:57.918682  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:57.934116  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 01:58:57.936846  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:57.962085  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:57.973665  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 01:58:57.994951  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:57.998319  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:58.034736  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:58.039659  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:58.057926  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:58.074678  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 01:58:58.198818  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:58.201898  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 01:58:58.219564  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 01:58:58.225744  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:58.226328  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 01:58:58.239886  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:58.246733  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:58.253036  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:58.258932  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:58.298753  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:58.318967  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:58.385217  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 01:58:58.445611  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 01:58:58.485457  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:58.499697  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:58.515390  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:58.536588  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:58.548015  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:58.572551  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:58.673930  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:58.703990  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 01:58:58.789402  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:58.810971  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:58.839257  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:58.854694  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:58.858618  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:58:58.871562  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 01:58:58.886307  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:58:58.959023  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:59.036469  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 01:58:59.066766  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:58:59.081769  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:59.101138  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:58:59.106314  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:59.133600  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:59.134253  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:59.190964  6402 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 77 similar messages]
I20240502 01:58:59.220907  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:59.291641  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:59.315292  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:59.328774  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:58:59.332738  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 01:58:59.343510  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:59.353947  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:59.367774  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 01:58:59.395202  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:58:59.415398  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 01:58:59.507511  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:58:59.511718  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 01:58:59.514552  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:58:59.659267  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 01:58:59.709420  6000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 01:58:59.715605  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 01:58:59.735221  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:59.735911  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:59.741616  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:59.750149  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:59.768200  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:59.811023  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 01:58:59.888684  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 01:58:59.890220  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:58:59.892341  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:58:59.902472  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:58:59.925755  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 01:58:59.932461  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:59.947378  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:58:59.976162  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:58:59.985344  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:00.017297  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:00.033844  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:00.195415  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:00.207078  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:00.217057  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:00.219197  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:00.219445  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 01:59:00.219642  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 01:59:00.236836  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:00.266896  6136 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240502 01:59:00.269814  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:00.385838  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 01:59:00.414111  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 01:59:00.457226  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:00.469226  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 01:59:00.484727  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:00.486209  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:00.491493  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:00.497253  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:00.500615  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:00.528285  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:00.540005  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:00.604199  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 01:59:00.608057  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 01:59:00.620239  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 01:59:00.744324  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:00.763667  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:00.785916  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:00.803194  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:00.820587  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:00.851634  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:00.870930  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 01:59:00.885183  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 01:59:00.926940  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 01:59:01.014976  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 01:59:01.023624  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:01.037933  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:01.084585  7348 raft_consensus.cc:482] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 01:59:01.084863  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:01.084831  7348 raft_consensus.cc:504] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:59:01.085642  7348 leader_election.cc:288] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 01:59:01.086602  6270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 01:59:01.088107  6270 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:01.088331  6269 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:01.088416  6269 leader_election.cc:302] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:59:01.088883  7348 raft_consensus.cc:2726] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:59:01.109033  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:01.160871  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:01.169549  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:01.182736  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 01:59:01.253784  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:01.269973  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:01.275228  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:01.284281  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 01:59:01.301666  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:01.307287  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:01.333426  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:01.333524  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:01.359813  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:01.511503  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:01.517154  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 01:59:01.598906  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:01.612092  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:01.628248  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:01.639127  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:01.647629  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 01:59:01.665176  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:01.682684  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:01.690841  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:01.836283  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:01.857767  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:01.869006  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:01.896313  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:01.916872  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:01.931636  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 01:59:01.945698  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:01.953662  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 01:59:01.994379  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:02.010901  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:02.016577  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 01:59:02.129729  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:02.149365  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:02.169772  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:02.174841  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:02.182708  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 01:59:02.182894  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 01:59:02.200485  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:02.255648  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 01:59:02.256968  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:02.367334  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:02.387558  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:02.424894  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:02.426474  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:02.446560  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:02.467909  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:02.482702  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:02.492960  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:02.513752  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 01:59:02.634625  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:02.642205  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:02.656260  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:02.668704  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 01:59:02.686151  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 01:59:02.722179  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:02.743569  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:02.788538  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:02.788829  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 01:59:02.870199  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 01:59:02.889981  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 01:59:02.957355  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:02.972965  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:02.973225  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:02.987056  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:03.025671  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:03.036232  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 01:59:03.052104  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 01:59:03.057075  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:03.071492  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:03.097470  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:03.118562  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 01:59:03.240667  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 01:59:03.251704  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:03.275276  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:03.281111  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:03.302229  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:03.311040  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 01:59:03.313081  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:03.315402  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 01:59:03.333184  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:03.380139  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:03.385685  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 01:59:03.388924  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 01:59:03.407739  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:03.408665  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:03.412665  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:03.434352  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:03.438760  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:03.472832  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:03.511626  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 01:59:03.540720  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:03.685596  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:03.692282  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:03.706286  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:03.712522  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:03.755451  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:03.776132  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:03.783211  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:03.797559  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 01:59:03.821579  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 01:59:03.918687  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:03.926003  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:03.942399  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:03.961473  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:04.004731  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:04.042346  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:04.060195  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:04.176699  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:04.193439  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:04.210001  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:04.210740  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:04.224295  6401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 01:59:04.227535  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 01:59:04.249903  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:04.278898  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:04.378897  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:04.406705  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:04.409763  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:04.435726  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:04.455281  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:04.465775  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:04.477365  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 01:59:04.486917  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 01:59:04.541430  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:04.581173  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 01:59:04.589046  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:04.646274  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 01:59:04.730160  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:04.733007  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:04.750098  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:04.806946  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:04.807257  6000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240502 01:59:04.808902  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 01:59:04.822685  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:04.843480  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:04.865494  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 01:59:04.918329  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:04.970613  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:05.030232  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 01:59:05.056682  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 01:59:05.063158  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:05.063656  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:05.087910  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 01:59:05.091650  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:05.118144  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:05.122326  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:05.122781  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:05.145833  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:05.164430  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:05.207358  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:05.216472  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:05.252517  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 01:59:05.253396  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:05.316859  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:05.357229  6136 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 01:59:05.359431  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 01:59:05.377483  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 01:59:05.384667  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:05.386032  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 01:59:05.419610  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 01:59:05.478364  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:05.481989  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:05.489306  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:05.500959  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:05.504254  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:05.505321  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:05.508159  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:05.592130  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 01:59:05.625713  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 01:59:05.769357  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:05.823935  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:05.825852  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:05.842065  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:05.845412  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 01:59:05.859344  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:05.864280  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:05.871345  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:05.874089  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 01:59:05.874224  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:05.922081  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:05.932206  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 01:59:05.946807  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:05.950559  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 01:59:05.971220  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:05.979668  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:06.034286  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:06.058292  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:06.073076  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 01:59:06.206288  6269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240502 01:59:06.279644  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:06.312361  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:06.313405  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:06.321933  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:06.331833  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:06.353201  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:06.363615  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:06.401373  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:06.401964  7980 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:06.402335  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:06.429651  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:06.462884  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:06.475967  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 01:59:06.495572  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:06.518970  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 01:59:06.526048  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:06.728848  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:06.748915  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:06.767907  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:06.794493  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 01:59:06.795428  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:06.822641  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:06.838486  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:06.838614  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 01:59:07.006758  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:07.045740  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:07.054970  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:07.085182  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:07.086656  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:07.092849  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 01:59:07.098660  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:07.103488  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 01:59:07.122838  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:07.123010  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:07.286238  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:07.287402  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 01:59:07.305907  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:07.314540  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:07.315573  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:07.365994  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:07.395629  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:07.399876  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 01:59:07.407132  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:07.416157  8453 raft_consensus.cc:482] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 01:59:07.416347  8453 raft_consensus.cc:504] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:59:07.417001  8453 leader_election.cc:288] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 01:59:07.419940  5999 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:07.475827  6000 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:07.476056  6000 leader_election.cc:302] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:59:07.476383  8453 raft_consensus.cc:2726] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:59:07.488292  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 01:59:07.528013  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 01:59:07.634020  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:07.641077  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:07.641193  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:07.648919  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:07.681159  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:07.700356  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 01:59:07.705580  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:07.735206  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:07.791162  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 01:59:07.802803  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 01:59:07.804040  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 01:59:07.819041  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:07.861315  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 01:59:07.876389  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:07.879379  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:07.920714  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:07.938463  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:07.960492  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:08.006075  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:08.072497  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 01:59:08.086597  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 01:59:08.142315  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 01:59:08.146755  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:08.160252  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:08.180040  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:08.193703  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:08.220508  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 01:59:08.223853  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:08.238140  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 01:59:08.256677  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:08.278468  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:08.279107  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 01:59:08.288651  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 01:59:08.304186  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 01:59:08.317441  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:08.389636  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:08.421854  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:08.439481  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:08.445928  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:08.458449  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:08.499191  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:08.570282  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 01:59:08.691406  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:08.709185  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:08.728657  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:08.748143  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:08.762393  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:08.774690  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 01:59:08.801050  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:08.824452  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:08.906112  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:08.945554  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 01:59:08.953078  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:08.963615  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:08.982002  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:09.005625  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:09.020923  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:09.054250  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:09.067484  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 01:59:09.144531  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:09.158636  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:09.180617  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:09.205489  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:09.211536  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:09.212040  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 01:59:09.229694  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:09.253258  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:09.264715  6401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 01:59:09.289428  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:09.515622  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 01:59:09.517853  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:09.528717  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 01:59:09.542135  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:09.551044  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:09.596300  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:09.642037  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:09.644744  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:09.649406  6402 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.133:43379, user_credentials={real_user=slave}} blocked reactor thread for 66275.5us
W20240502 01:59:09.666756  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:09.824836  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:09.848079  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:09.860038  6000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240502 01:59:09.862157  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:09.870417  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:09.874754  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:09.886034  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 01:59:09.897367  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:09.904906  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:09.906877  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 01:59:09.910763  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:09.932796  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:09.949532  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:09.987491  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:10.024783  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 01:59:10.031459  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:10.090607  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:10.204281  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 01:59:10.216286  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:10.314739  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 01:59:10.319722  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 01:59:10.344748  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:10.375265  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:10.376471  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:10.391393  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 01:59:10.415480  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:10.418936  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:10.436878  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:10.446135  6135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 01:59:10.469328  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 01:59:10.470824  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:10.498113  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 01:59:10.505012  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 01:59:10.522958  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 01:59:10.610252  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:10.621263  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:10.654300  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:10.658540  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:10.670913  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:10.692016  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:10.703541  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:10.706879  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 01:59:10.754905  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 01:59:10.844358  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 01:59:10.876989  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:10.887656  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 01:59:10.899101  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 01:59:10.916221  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:10.927717  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:10.932545  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 01:59:10.955960  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:10.970943  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 01:59:11.004720  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:11.005673  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:11.122279  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:11.212278  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:11.233863  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:11.254969  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:11.255394  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:11.256243  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:11.287451  6270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240502 01:59:11.306499  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:11.323559  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 01:59:11.325013  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:11.371338  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 01:59:11.391137  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:11.414549  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:11.416893  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 01:59:11.421047  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:11.421993  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:11.455663  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:11.457321  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:11.473250  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:11.627836  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 01:59:11.675302  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:11.688558  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:11.702678  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:11.745133  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:11.764360  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:11.815474  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:11.829244  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:11.834332  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:11.835834  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 01:59:11.882315  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:11.906637  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:11.927261  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:11.974817  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:11.984230  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 01:59:12.026309  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:12.046567  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:12.168313  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:12.188212  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:12.212960  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:12.233273  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 01:59:12.234923  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:12.236248  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:12.239753  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:12.293656  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:12.391178  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 01:59:12.425047  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:12.433954  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:12.443961  9160 raft_consensus.cc:482] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 01:59:12.444170  9160 raft_consensus.cc:504] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:59:12.444944  9160 leader_election.cc:288] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 01:59:12.448892  5999 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:12.449234  6000 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:12.449362  6000 leader_election.cc:302] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:59:12.449945  9160 raft_consensus.cc:2726] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 01:59:12.465410  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:12.498847  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:12.502919  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:12.504623  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:12.507568  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:12.612381  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 01:59:12.669961  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:12.736148  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:12.737529  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 01:59:12.764817  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:12.777596  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:12.788399  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:12.790725  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 01:59:12.822197  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:12.842456  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:12.877228  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 01:59:12.887643  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 01:59:12.890056  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 01:59:12.915272  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:12.937731  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 01:59:12.975703  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:13.006780  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 01:59:13.010416  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 01:59:13.011374  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:13.020674  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:13.060431  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:13.083590  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:13.084556  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:13.205499  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 01:59:13.229609  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:13.286350  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:13.299263  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:13.304176  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:13.321719  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:13.341572  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:13.355397  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:13.364246  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 01:59:13.401376  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:13.411984  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 01:59:13.417924  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 01:59:13.425304  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 01:59:13.440119  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:13.463447  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:13.491081  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:13.506912  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 01:59:13.511530  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:13.555153  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:13.606657  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:13.609370  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:13.746131  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 01:59:13.794833  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:13.821763  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:13.823369  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:13.823408  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:13.837200  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:13.854640  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:13.863664  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 01:59:13.874542  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 01:59:13.908397  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:13.998138  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:14.025547  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 01:59:14.025632  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:14.043797  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:14.070166  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:14.091612  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:14.105494  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:14.147339  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 01:59:14.220783  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:14.233839  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:14.261914  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:14.280994  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:14.292728  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 01:59:14.309422  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:14.310089  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:14.323380  6402 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240502 01:59:14.332641  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:14.357496  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:14.542619  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:14.564080  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:14.589308  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:14.604317  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:14.643515  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:14.644071  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 01:59:14.660806  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 01:59:14.675226  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:14.759068  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:14.764783  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 01:59:14.787276  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:14.792122  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:14.798139  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:14.808585  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:14.821774  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:14.882489  6000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 72 similar messages]
I20240502 01:59:14.893040  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:14.944408  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:15.013887  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 01:59:15.064247  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:15.099000  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:15.111534  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:15.125727  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:15.160048  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:15.164958  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:15.180446  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:15.254655  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:15.268891  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:15.269587  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:15.287433  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:15.292368  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:15.299355  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:15.317258  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 01:59:15.329788  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 01:59:15.333982  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 01:59:15.370324  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 01:59:15.388773  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 01:59:15.407548  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:15.489260  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 01:59:15.503122  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 01:59:15.518527  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:15.527446  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:15.530462  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:15.538986  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:15.550805  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:15.556087  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:15.691527  6135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 01:59:15.694325  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:15.749903  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 01:59:15.765020  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 01:59:15.803808  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 01:59:15.814090  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 01:59:15.817338  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:15.829367  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:15.860128  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 01:59:15.862010  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:15.897545  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:15.920293  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:15.921602  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:16.015758  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:16.032531  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:16.045432  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:16.067926  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:16.111615  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 01:59:16.118726  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:16.122923  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:16.148524  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 01:59:16.150173  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:16.153378  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:16.193150  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 01:59:16.240213  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 01:59:16.302724  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:16.310900  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:16.335431  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:16.337257  6269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 52 similar messages]
I20240502 01:59:16.338940  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:16.357785  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:16.408164  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 01:59:16.427506  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:16.447345  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 01:59:16.533320  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:16.582158  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:16.585659  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:16.612488  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:16.613633  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:16.670320  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:16.685623  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:16.687418  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 01:59:16.839991  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 01:59:16.847388  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:16.860975  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:16.861071  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:16.876152  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:16.900358  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:16.931394  5998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.135:40543, user_credentials={real_user=slave}} blocked reactor thread for 47927.9us
I20240502 01:59:16.940429  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:16.951742  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:16.978716  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:17.106470  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:17.114022  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 01:59:17.172809  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:17.198073  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:17.211781  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:17.214985  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 01:59:17.246711  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:17.256110  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 01:59:17.292284  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:17.292757  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:17.465315  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:17.488265  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:17.493902  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:17.496310  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:17.522887  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:17.550537  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:17.557535  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:17.614128  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:17.670862  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 01:59:17.747673  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:17.769100  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:17.781960  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:17.805609  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:17.808375  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 01:59:17.831617  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 01:59:17.847282  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:17.864498  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 01:59:17.870429  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:17.948210  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 01:59:17.970269  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 01:59:17.980190  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 01:59:18.025884  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:18.042605  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:18.046938  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:18.056341  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:18.060842  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 01:59:18.102250  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:18.127379  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:18.133009  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 01:59:18.146765  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:18.170511  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:18.197484  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 01:59:18.234169  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:18.245654  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:18.263996  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:18.309156  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:18.335206  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:18.342427  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:18.356760  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 01:59:18.363726  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 01:59:18.380153  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 01:59:18.447602  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 01:59:18.469564  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:18.549433  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:18.563835  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:18.589177  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:18.594033  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:18.599176  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:18.608776  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 01:59:18.617563  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:18.649865  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 01:59:18.688347  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:18.769608  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:18.807731  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:18.817912  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:18.821478  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:18.847746  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:18.870859  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:18.872006  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:18.901482  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 01:59:18.927542  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 01:59:18.930517 10228 raft_consensus.cc:482] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 01:59:18.930696 10228 raft_consensus.cc:504] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:59:18.931316 10228 leader_election.cc:288] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 01:59:18.933593  6270 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:18.935331  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 01:59:18.937369  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 01:59:18.947934  6269 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:18.948091  6269 leader_election.cc:302] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:59:18.948567 10228 raft_consensus.cc:2726] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 01:59:19.045085  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:19.048741  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:19.055979  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:19.092919  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:19.117125  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:19.142714  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:19.206566  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 01:59:19.258723  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:19.331694  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:19.345008  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:19.349866  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:19.354722  6401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240502 01:59:19.355782  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:19.360730  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:19.394781  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 01:59:19.403239  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:19.441123  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:19.446377  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 01:59:19.629395  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:19.647929  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:19.658635  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:19.673462  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:19.685504  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:19.713352  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:19.779314  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:19.813091  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 01:59:19.845329  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:19.889569  6000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 68 similar messages]
I20240502 01:59:19.933753  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:19.951614  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:19.972563  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:19.973068  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:20.001623  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:20.019511  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:20.133217  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:20.180248  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:20.193338  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:20.214859  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:20.230588  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:20.234856  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 01:59:20.244567  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 01:59:20.256273  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:20.259236  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 01:59:20.274931  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:20.287596  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:20.295022  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:20.327226  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:20.333945  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:20.368319  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:20.410650  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:20.421581  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 01:59:20.436220  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:20.449733  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 01:59:20.459015  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 01:59:20.520613  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 01:59:20.555132  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 01:59:20.655488  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:20.672757  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:20.686419  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:20.698845  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:20.708707  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:20.713009  6135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 01:59:20.714185  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 01:59:20.752347  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:20.764225  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:20.785072  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:20.879228  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:20.894937  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 01:59:20.908823  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 01:59:20.912922  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 01:59:20.923667  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:20.949141  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:20.956702  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 01:59:20.968641  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:20.969710  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:20.973637  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:21.083345  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:21.103018  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 01:59:21.108250  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:21.132218  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:21.143002  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:21.144379  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 01:59:21.181221  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:21.182878  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:21.199143  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:21.306123  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 01:59:21.349694  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 01:59:21.397435  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:21.406473  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:21.408550  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:21.413362  6270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240502 01:59:21.415352  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 01:59:21.422618  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 01:59:21.430984  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:21.437247  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:21.463431  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:21.487710  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:21.625327  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:21.644379  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:21.677789  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:21.688812  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 01:59:21.705693  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:21.725232  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:21.753085  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 01:59:21.797633  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:21.799472  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:21.889000  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:21.972031  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:21.976256  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:21.978672  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:21.981220  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:21.997628  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:22.045276  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:22.072084  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 01:59:22.073909  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 01:59:22.106983  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:22.158488  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:22.185063  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:22.193850  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:22.197773  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:22.231462  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:22.330638  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:22.330870  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:22.381353  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 01:59:22.417881  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:22.418700  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:22.432844  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:22.433826  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:22.440373  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:22.455466  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:22.651225  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 01:59:22.652710  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 01:59:22.762907  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:22.767483  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:22.768596  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 01:59:22.786175  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:22.802155  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:22.805632  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:22.816092  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:22.819988  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:22.829245  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 01:59:22.829736  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:22.845358  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:22.858857  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:22.860868  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:22.861517  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:22.866417  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:22.910841  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:22.941463  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 01:59:22.988737  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 01:59:23.131876  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 01:59:23.132064  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:23.138020  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:23.158540  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:23.161643  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:23.189181  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 01:59:23.196058  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:23.196523  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:23.215790  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 01:59:23.220584  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 01:59:23.251991  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:23.323679  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 01:59:23.354425  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:23.376348  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:23.383388  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:23.401252  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 01:59:23.410542  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:23.439291  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 01:59:23.448563  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:23.450062  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 01:59:23.500034  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:23.538398  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:23.563053  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 01:59:23.664355  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 01:59:23.681167  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 01:59:23.683655  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:23.691990  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:23.711562  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 01:59:23.716500  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:23.722018  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:23.745721  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:23.768241  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:23.832366  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:23.908131  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:24.043270  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:24.064812  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 01:59:24.084501  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:24.090035  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:24.109997  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:24.152751  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:24.216138  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:24.232540  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 01:59:24.234558  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:24.255075  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:24.257831  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:24.270623  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:24.300767  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:24.336799  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:24.360991  6401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240502 01:59:24.388468  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:24.415896  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 01:59:24.439787  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:24.522331  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 01:59:24.556483  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:24.563314  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:24.582307  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:24.609228  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:24.653545  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:24.692798  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:24.695286  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 01:59:24.768011  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:24.823709  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 01:59:24.859364  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:24.866333  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:24.867317  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:24.890630  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:24.892123  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:24.929638  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:24.939489  5999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240502 01:59:24.941030  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 01:59:25.022805  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:25.075498  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:25.094638  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:25.107092  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:25.146930  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:25.171276  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:25.209367  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:25.213290  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:25.284780  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 01:59:25.409286  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:25.417502  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:25.418437  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:25.432116  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:25.432511  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 01:59:25.433616  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 01:59:25.452901  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 01:59:25.453243  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:25.458196  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:25.498924  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:25.563848  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:25.592221  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:25.603538  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:25.621068  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:25.622159  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 01:59:25.643072  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:25.653461  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 01:59:25.685387  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 01:59:25.688519  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 01:59:25.697566  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:25.716953  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 01:59:25.723831  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:25.774106  6135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 01:59:25.775139  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 01:59:25.811441  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 01:59:25.831213  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 01:59:25.880641  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:25.898370  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 01:59:25.899096  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:25.899401  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:25.899853  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:25.923825  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:25.952003  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:25.972360  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 01:59:26.034483  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 01:59:26.058871  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:26.075167  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 01:59:26.091419  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:26.110579  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 01:59:26.196098  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:26.202821  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:26.233951  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:26.262074  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:26.289518  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:26.291378  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:26.311214  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:26.322212  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:26.338197  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:26.350142  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:26.373344  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:26.424239  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:26.485050  6269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 01:59:26.626183  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:26.671046  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:26.673550  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:26.692113  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:26.697647  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:26.699146  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:26.729974  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:26.746645  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:26.766356  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:26.772416  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:26.783969  6134 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.135:40543, user_credentials={real_user=slave}} blocked reactor thread for 72820.2us
W20240502 01:59:26.798700  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 01:59:26.821133  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:26.835196  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:26.864238  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:26.901629  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:26.927191  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 01:59:26.948246  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 01:59:26.958346  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 01:59:27.123034  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 01:59:27.167901  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:27.172482  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:27.184154  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:27.195350  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:27.205888  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:27.207466  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:27.219287  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:27.235563  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:27.255401  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:27.282670  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:27.286808  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:27.318400  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:27.368964  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:27.404847  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:27.429415  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 01:59:27.554148  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:27.570288  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:27.589148  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:27.623590  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 01:59:27.623740  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:27.653795  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:27.654073  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:27.685870  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:27.709991  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 01:59:27.730240  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:27.807502  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 01:59:27.869802  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:27.885313  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:27.900082  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:27.921667  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:27.965490  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:28.024780  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 01:59:28.030092  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:28.030093  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 01:59:28.039559  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 01:59:28.066300  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 01:59:28.083629  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 01:59:28.100472  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 01:59:28.124511  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 01:59:28.140245  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 01:59:28.200539  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:28.204669  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:28.237156  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:28.249596  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 01:59:28.251528  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:28.283917  6271 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.130:38197, user_credentials={real_user=slave}} blocked reactor thread for 57226.3us
I20240502 01:59:28.296146  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:28.296245  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 01:59:28.298362  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:28.322037  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:28.322540  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:28.422207  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 01:59:28.455142  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 01:59:28.478739  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:28.488525  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:28.491051  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 01:59:28.493134  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:28.508057  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:28.508685  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:28.512480  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:28.545441  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 01:59:28.568492  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 01:59:28.714208  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:28.730015  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:28.738943  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:28.749413  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:28.770391  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:28.798568  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:28.850284  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:28.854607  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:29.025100  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:29.037540  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:29.038152  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:29.055961  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 01:59:29.056974  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:29.084373  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:29.129987  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:29.188720  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 01:59:29.261505  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:29.281723  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:29.299978  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:29.300992  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:29.340574  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:29.376753  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:29.377048  6401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240502 01:59:29.399672  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:29.435916  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:29.530404  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:29.536548  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:29.542042  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 01:59:29.546651  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:29.562877  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:29.563768  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:29.600237  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 01:59:29.601652  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:29.631642  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 01:59:29.823077  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:29.848786  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:29.853305  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:29.857627  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:29.882294  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:29.907019  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:29.919193  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:29.945133  5999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240502 01:59:29.948381  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 01:59:29.950028  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:30.021592  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 01:59:30.082036  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:30.095789  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:30.160949  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:30.165684  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:30.179567  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:30.202383  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:30.204231  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:30.224737  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:30.249773  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 01:59:30.253196  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:30.263425  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:30.297641  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:30.325093  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:30.329865  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:30.381966  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:30.389120  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 01:59:30.389258  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 01:59:30.438164  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 01:59:30.472565  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 01:59:30.480577  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 01:59:30.486657  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 01:59:30.503041  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 01:59:30.553669  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:30.576685  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:30.592694  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:30.613924  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:30.636621  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:30.670755  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:30.687583  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 01:59:30.695529  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 01:59:30.751750  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 01:59:30.784713  6135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 01:59:30.785835  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 01:59:30.802001  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 01:59:30.821278  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:30.828824  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:30.857555  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:30.861898  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:30.875733  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:30.888424  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:30.891387  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 01:59:30.902945  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:30.910568  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 01:59:30.919461  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:31.042323  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 01:59:31.152662  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:31.173933  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:31.197604  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:31.225088  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 01:59:31.227181  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:31.233693  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:31.268024  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:31.303999  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:31.414238  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:31.414785  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:31.472829  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:31.482354  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:31.496881  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:31.516043  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:31.543711  6269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240502 01:59:31.577071  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:31.624548  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:31.630362  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:31.645583  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:31.667331  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 01:59:31.679767  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:31.702647  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:31.708015  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:31.708216  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:31.802799  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 01:59:31.959100  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:31.979619  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 01:59:31.980636  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:31.989567 12111 raft_consensus.cc:482] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 01:59:31.989770 12111 raft_consensus.cc:504] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:59:31.990541 12111 leader_election.cc:288] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 01:59:31.995033  5999 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:32.030088  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:32.046677  6000 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:32.047034  6000 leader_election.cc:302] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:59:32.047405 12111 raft_consensus.cc:2726] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 01:59:32.048738  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:32.059733  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:32.061901  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:32.069407  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:32.121419  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 01:59:32.166213  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 01:59:32.207682  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:32.274406  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:32.291081  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:32.298496  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:32.334378  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:32.349690  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:32.361851  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 01:59:32.384606  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:32.432143  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 01:59:32.517163  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:32.536238  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:32.553738  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:32.556073  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:32.572535  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:32.588858  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:32.617069  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:32.702955  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 01:59:32.772609  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 01:59:32.789775  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:32.806167  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 01:59:32.895618  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:32.896427  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 01:59:32.902041  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 01:59:32.927275  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:32.932757  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:32.957640  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:32.988168  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:33.004093 12128 raft_consensus.cc:482] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 01:59:33.004343 12128 raft_consensus.cc:504] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:59:33.005231 12128 leader_election.cc:288] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 01:59:33.010735  6269 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:33.011682  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:33.012696  6270 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:33.012835  6270 leader_election.cc:302] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:59:33.013248  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 01:59:33.013696 12128 raft_consensus.cc:2726] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 01:59:33.023507  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:33.033221  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:33.053970  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 01:59:33.054299  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 01:59:33.062842  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:33.063074  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:33.077498  6402 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.133:43379, user_credentials={real_user=slave}} blocked reactor thread for 133416us
W20240502 01:59:33.088611  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 01:59:33.102906  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:33.125203  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:33.153738  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:33.173786  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 01:59:33.181555  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:33.279289  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 01:59:33.347992  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 01:59:33.391052  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:33.400645  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 01:59:33.429106  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:33.437886  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 01:59:33.443671  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:33.476668  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:33.503765  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:33.537930  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:33.576810  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 01:59:33.591118  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:33.630463  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:33.679461  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 01:59:33.680065  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:33.685773  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:33.709470  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 01:59:33.711104  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:33.747931  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:33.756404  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:33.765338  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:33.922446  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:33.949651  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:33.986604  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:33.990689  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:34.015333  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:34.041826  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:34.244818  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:34.247155  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:34.260200  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:34.260843  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:34.266752  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:34.271281  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:34.280236  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:34.295475  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:34.428365  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 01:59:34.435732  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 01:59:34.445302  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 01:59:34.509919  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:34.536621  6401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 79 similar messages]
I20240502 01:59:34.558632  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:34.561743  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:34.576123  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:34.591301  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 01:59:34.593267  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:34.650218  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:34.806164  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 01:59:34.814622  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:34.826740  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:34.831842  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:34.840265  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:34.875337  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 01:59:34.877061  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:34.885753  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:34.886754  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:34.909847  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:34.920271  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:34.920874  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 01:59:34.944620  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:34.966665  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:34.988054  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:35.028388  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:35.029954  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 01:59:35.032119  6000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240502 01:59:35.049288  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:35.086638  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 01:59:35.202718  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 01:59:35.259745  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:35.277755  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:35.292348  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:35.293536  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:35.298736  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:35.325373  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:35.373528  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:35.432722  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 01:59:35.435703  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:35.445847  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:35.450317  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:35.451555  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:35.458810  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:35.462528  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 01:59:35.530369  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:35.538743  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 01:59:35.544000  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 01:59:35.547053  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:35.600687  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 01:59:35.675454  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 01:59:35.675810  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 01:59:35.699165  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 01:59:35.738196  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:35.738369  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:35.739282  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:35.746667  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:35.749329  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:35.755196  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:35.782444  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:35.787616  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 01:59:35.840270  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 01:59:35.885174  6135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 01:59:35.913879  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 01:59:35.956475  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 01:59:36.003286  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:36.101428  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:36.106040  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:36.113704  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:36.126154  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:36.135685  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:36.168807  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:36.240314  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:36.248404  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 01:59:36.275486  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:36.279309  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:36.297996  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:36.315083  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:36.315150  6268 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.135:40543, user_credentials={real_user=slave}} blocked reactor thread for 45497.7us
I20240502 01:59:36.334589  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:36.345336  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 01:59:36.421605  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:36.505124  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:36.578876  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:36.597499  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:36.603631  6269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 53 similar messages]
I20240502 01:59:36.610272  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:36.621346  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:36.666170  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 01:59:36.671218  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:36.680176  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:36.731287  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 01:59:36.755349  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:36.856657  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 01:59:36.950758  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:36.974072  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:36.988868  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:36.995836  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:37.011334  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:37.045744  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:37.066349  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 01:59:37.120349  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:37.214798  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:37.228828  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:37.250038  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:37.281286  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:37.282397  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:37.302614  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 01:59:37.321223  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 01:59:37.330917  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:37.334210  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:37.468283  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 01:59:37.516919  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:37.538594  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:37.556694  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 01:59:37.565627  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:37.574303  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:37.577992  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:37.579062  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:37.601850  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 01:59:37.626322  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:37.677738  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:37.747611  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 01:59:37.803774  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:37.836717  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:37.850548  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 01:59:37.858527  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:37.887194  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:37.887650  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:37.891281  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:37.896528  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 01:59:37.968081  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 01:59:37.997606 12968 raft_consensus.cc:482] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 01:59:37.997814 12968 raft_consensus.cc:504] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:59:37.998598 12968 leader_election.cc:288] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 01:59:38.000541  6000 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:38.001502  5999 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:38.001647  5999 leader_election.cc:302] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:59:38.002166 12968 raft_consensus.cc:2726] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:59:38.061338  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 01:59:38.105206  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:38.128998  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:38.130107  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:38.134191  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 01:59:38.135442  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:38.135777  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 01:59:38.206292  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 01:59:38.226804  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:38.227594  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:38.250957  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:38.281838  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:38.344429  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 01:59:38.385782  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 01:59:38.397698  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 01:59:38.441820  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:38.461697  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:38.463097  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:38.484625  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:38.497696  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:38.521420  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:38.538987  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:38.542150  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 01:59:38.548633  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:38.573745  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:38.578198  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:38.578712  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:38.611493  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 01:59:38.613037  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:38.634836  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:38.677714  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 01:59:38.742463  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 01:59:38.795789  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:38.876595  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:38.884119  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:38.889518  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:38.916841  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:38.947847  6268 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.132:36867, user_credentials={real_user=slave}} blocked reactor thread for 52249.5us
I20240502 01:59:39.022094  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:39.051119  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:39.059348  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:39.085598  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:39.086647  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:39.087693  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:39.106300  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:39.183024  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:39.191596  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:39.277735  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 01:59:39.314204  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:39.318692  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:39.328306  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:39.332356  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:39.337610  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:39.345140  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:39.353334  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 01:59:39.372220  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:39.476142  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 01:59:39.481598  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:39.571544  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 01:59:39.601649  6401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240502 01:59:39.602425  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:39.618237  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:39.625144  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:39.685062  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:39.700742  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:39.726562  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:39.770671  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 01:59:39.784037  6134 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.135:40543, user_credentials={real_user=slave}} blocked reactor thread for 76263.1us
I20240502 01:59:39.800065  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:39.821928  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:39.854710  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:39.863727  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:39.877105  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:39.880204  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:39.918689  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:39.936134  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:39.947330  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 01:59:39.962327  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 01:59:40.078408  5999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240502 01:59:40.079829  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 01:59:40.086184  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 01:59:40.153962  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 01:59:40.210352  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:40.213924  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:40.231700  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:40.234385  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:40.236483  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:40.257728  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:40.274538  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:40.284013  6270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.133:43379, user_credentials={real_user=slave}} blocked reactor thread for 48049.4us
W20240502 01:59:40.300762  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 01:59:40.305169  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 01:59:40.380620  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:40.455045  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:40.470258  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:40.495640  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:40.501907  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:40.521823  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:40.533203  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:40.548120  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 01:59:40.586148  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:40.630874  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:40.632232  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:40.636287  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 01:59:40.666054  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:40.668845  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:40.680399  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 01:59:40.692272  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 01:59:40.704192  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:40.708441  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:40.758293  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:40.823969  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 01:59:40.829080  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 01:59:40.943121  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:40.968309  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:40.981249  6135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240502 01:59:40.998709  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:41.015272  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:41.015635  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:41.027371  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 01:59:41.033432  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:41.046182  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 01:59:41.055959  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 01:59:41.135413  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 01:59:41.145735  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:41.166630  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 01:59:41.174860  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:41.190971  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:41.200285  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 01:59:41.219173  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:41.231256  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:41.266798  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:41.270020  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:41.270182  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:41.448663  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:41.475236  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:41.518105  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:41.524245  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:41.528349  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:41.535720  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:41.637109  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:41.766702  6269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240502 01:59:41.782600  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:41.807423  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:41.820287  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:41.825955  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:41.837499  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:41.859108  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:41.860559  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:41.863806  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 01:59:41.879173  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 01:59:42.005949  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:42.022814  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 01:59:42.062036  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:42.063275  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:42.073575  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:42.091610  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:42.100531  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:42.125473  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:42.143805  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 01:59:42.204118  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 01:59:42.314638  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:42.335598  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:42.335642  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 01:59:42.337674  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:42.363412  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:42.390892  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:42.415143  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:42.425988  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 01:59:42.442173  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:42.472167  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 01:59:42.523109  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 01:59:42.526999  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:42.633930  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:42.634130  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:42.644418  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:42.652616  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:42.668362  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:42.670017  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:42.737084  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 01:59:42.765758  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:42.790421  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 01:59:42.808372  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:42.818356  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:42.828351  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:42.854131  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:42.855410  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:42.871745  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:42.918864  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 01:59:42.925552  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:42.975054  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 01:59:42.989149  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 01:59:43.083570  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:43.085130  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:43.093868  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:43.098320  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:43.112821  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:43.141204  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 01:59:43.162382  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:43.238165  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 01:59:43.278162  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:43.289408  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 01:59:43.307936  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:43.365429  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 01:59:43.407677  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 01:59:43.417840  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:43.419428  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:43.429405  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:43.445591  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:43.450428  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:43.466260  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 01:59:43.469106  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:43.492496  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 01:59:43.629302  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 01:59:43.639501  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 01:59:43.656415  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 01:59:43.704574  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:43.742301  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:43.791719  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:43.792709  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:43.827190  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:43.875085  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:43.886219  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:43.897581  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:44.069056  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:44.087941  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:44.089531  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:44.094848  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:44.114912  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:44.116492  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:44.169736  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:44.182575  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:44.194623  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:44.204701  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:44.226109  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:44.252732  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 01:59:44.258890  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:44.310734  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:44.393460  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 01:59:44.410418  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:44.500942  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 01:59:44.539289  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:44.557665  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:44.579468  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:44.602078  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 01:59:44.603178  6401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 01:59:44.611009  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 01:59:44.624140  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:44.626317  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:44.627251  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:44.669734  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:44.714087  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 01:59:44.752203  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:44.772758  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:44.802793  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 01:59:44.809645  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:44.837894  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:44.846662  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:44.849915  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:44.887190  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:44.953787  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 01:59:44.982874  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 01:59:44.991222  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 01:59:45.051240  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:45.077607  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:45.082376  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:45.085938  6000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240502 01:59:45.089740  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:45.096243  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:45.115585  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:45.116546  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:45.267901  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 01:59:45.321318  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:45.341862  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:45.345696  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:45.345889  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:45.383849  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:45.403970  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 01:59:45.406389  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 01:59:45.431254  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:45.434010  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:45.532379  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:45.532783  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:45.557557  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:45.577370  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:45.613713  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 01:59:45.628836  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:45.646459  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:45.677014  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:45.763573  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:45.764097  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 01:59:45.814898  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 01:59:45.825855  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 01:59:45.873988  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:45.899819  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:45.902832  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:45.946211  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:45.972714  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:45.978679  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:45.981292  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 01:59:45.990765  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 01:59:45.990936  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 01:59:46.017233  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:46.033552  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:46.049907  6316 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:46.049940  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:46.051813  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:46.054998  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 01:59:46.092530  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:46.108822  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 01:59:46.108997  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:46.110983  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 01:59:46.186350  6135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 01:59:46.197291  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:46.282447  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:46.297152  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 01:59:46.298837  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:46.301683  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:46.304267  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:46.329866  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:46.348057  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:46.386744  6270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.133:43379, user_credentials={real_user=slave}} blocked reactor thread for 96214.3us
I20240502 01:59:46.602653  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:46.613750  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:46.626994  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:46.647401  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:46.674002  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:46.695003  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 01:59:46.702752  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:46.730751  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 01:59:46.737572  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:46.752036  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:46.845039  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:46.873044  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:46.908089  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:46.920383  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:46.920759  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 01:59:46.935669  6270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 01:59:46.937263  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 01:59:46.937312  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:46.947052  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:47.100767  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:47.111791  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:47.136997  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:47.140004  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:47.185125  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:47.185664  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:47.194499  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 01:59:47.233742  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:47.274192  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 01:59:47.275784  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:47.293217  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 01:59:47.366494  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:47.397418  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:47.399289  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:47.416507  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:47.433423  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:47.488557  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:47.492116  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 01:59:47.567062  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:47.568862  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:47.570909  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:47.574709  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 01:59:47.579854  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:47.599793  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:47.618438  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 01:59:47.637522  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:47.653024  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 01:59:47.734982  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:47.807303  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:47.897429  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:47.928126  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:47.934679  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:47.949875  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 01:59:47.950166  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:47.961535  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:48.021720  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:48.068442  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:48.091024  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:48.139302  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:48.146457  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:48.174281  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 01:59:48.193653  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:48.193789  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 01:59:48.216820  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:48.285356  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 01:59:48.286340  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:48.293171  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 01:59:48.336315  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:48.392256  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:48.408242  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:48.411417  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:48.417621  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 01:59:48.470257  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:48.490089  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:48.514328  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:48.519750  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 01:59:48.585675  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 01:59:48.594601  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 01:59:48.597736  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 01:59:48.672964  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 01:59:48.697567  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:48.704252  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:48.708794  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 01:59:48.709339  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:48.740473  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:48.744542  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:48.759433  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 01:59:48.805725  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:48.820943  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:48.862859  6135 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.136:44359, user_credentials={real_user=slave}} blocked reactor thread for 189898us
W20240502 01:59:48.885600  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:48.945823 14606 raft_consensus.cc:482] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 01:59:48.946020 14606 raft_consensus.cc:504] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:59:48.946743 14606 leader_election.cc:288] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 01:59:48.949965  6270 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:48.967836  6269 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:48.968185  6269 leader_election.cc:302] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:59:48.968652 14606 raft_consensus.cc:2726] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 01:59:49.013271  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:49.045342  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 01:59:49.054258  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:49.058377  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:49.153254  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:49.156175  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:49.184207  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:49.292341  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:49.320626  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:49.354331  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:49.364079  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:49.379181  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:49.387725  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:49.399622  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 01:59:49.402431  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:49.408970  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 01:59:49.417060  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:49.486267  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 01:59:49.552101  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:49.563599  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:49.583101  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:49.588454  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:49.616557  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:49.617010  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:49.622562  6402 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240502 01:59:49.654707  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 01:59:49.739420  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 01:59:49.844877  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:49.854207  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:49.866298  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:49.867578  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:49.882114  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:49.887957  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:49.922349  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:49.967432  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:50.039093  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 01:59:50.041797  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:50.056576  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:50.064332  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 01:59:50.074013  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 01:59:50.083845  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:50.085309  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:50.109272  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:50.112658  6000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 67 similar messages]
I20240502 01:59:50.129845  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:50.264688  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 01:59:50.276772  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:50.277285  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:50.288431  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:50.288532  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:50.294878  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:50.314491  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:50.329463  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:50.413791  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 01:59:50.455564  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:50.523118  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 01:59:50.653546  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 01:59:50.685257  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 01:59:50.716704  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:50.725906  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:50.733362  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:50.736009  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 01:59:50.754799  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:50.771548  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:50.782063  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:50.788714  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:50.812642  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:50.839175  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:50.852679  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:50.910547  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 01:59:50.911614  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 01:59:50.927456  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:50.941840  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:50.943511  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:50.945148  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:50.982686  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:50.987500  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:51.010006  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:51.020845  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:51.021930  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 01:59:51.034811  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 01:59:51.040627  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:51.055529  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:51.139714  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 01:59:51.144927  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 01:59:51.155455  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:51.174930  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 01:59:51.325464  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:51.325469  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:51.326135  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 01:59:51.340169  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:51.341634  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:51.346607  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:51.347422  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:51.420289  6135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 01:59:51.451618  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:51.619241  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:51.635651  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 01:59:51.656985  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 01:59:51.717968  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:51.740837  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:51.766606  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:51.809708  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:51.824911  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 01:59:51.833276  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:51.838210  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:51.987792  6269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 01:59:51.989456  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 01:59:52.000643  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 01:59:52.006986  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:52.033138  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:52.052960  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:52.069150  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:52.087745  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:52.101831  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:52.117316  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:52.120800  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 01:59:52.144245  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:52.156029  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:52.156605  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:52.176697  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:52.188205  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:52.207757  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:52.248546  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:52.496714  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 01:59:52.510751  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:52.510851  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:52.527235  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:52.557778  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:52.567097  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 01:59:52.579844  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:52.580876  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:52.672976  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 01:59:52.680490  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:52.699364  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 01:59:52.702648  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 01:59:52.708465  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:52.710522  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:52.730422  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:52.744721  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:52.757670  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:52.773376  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:52.789194  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:52.966277  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:52.990064  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:53.004978  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:53.037495  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:53.043747  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:53.044585  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 01:59:53.046062  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 01:59:53.054641  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:53.080421  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 01:59:53.098909  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 01:59:53.100630  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:53.130824  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 01:59:53.226158  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 01:59:53.233777  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:53.238839  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:53.260165  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:53.260344  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:53.296823  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 01:59:53.297675  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:53.326478  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:53.328696  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:53.345455  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 01:59:53.502490  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:53.525429  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:53.526774  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:53.527204  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 01:59:53.531836  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:53.566082  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 01:59:53.579715  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:53.585444  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 01:59:53.594166  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:53.617772  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:53.701818  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:53.867683  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:53.879293  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 01:59:53.881160  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 01:59:53.892304  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:53.921717  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:53.945443  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:54.008994  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:54.054081  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:54.055622  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:54.154623  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:54.185529  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:54.190338  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:54.203763  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:54.205521  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:54.250298  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:54.267680  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 01:59:54.269100  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:54.291193  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:54.295114  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:54.308815  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:54.324649  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:54.347671  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:54.348372  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:54.394198  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 01:59:54.428853  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 01:59:54.515992  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 01:59:54.592255  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 01:59:54.624353  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:54.633301  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:54.634270  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:54.642153  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:54.652757  6402 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 79 similar messages]
I20240502 01:59:54.668380  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:54.699398  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:54.705262  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:54.730161  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 01:59:54.748695  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:54.795326  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:54.839249  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:54.868278  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:54.875715  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:54.892359  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:54.922331  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:55.032394  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 01:59:55.079134  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 01:59:55.090096  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:55.097543  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:55.115744  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:55.142598  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:55.146755  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:55.160904  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:55.161556 15211 raft_consensus.cc:482] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 01:59:55.161725 15211 raft_consensus.cc:504] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:59:55.162482 15211 leader_election.cc:288] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 01:59:55.169859  6000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 01:59:55.171504  6269 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:55.171805  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 01:59:55.220690  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:55.229074  6270 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:55.229243  6270 leader_election.cc:302] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:59:55.229519 15211 raft_consensus.cc:2726] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:59:55.250295  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 01:59:55.275571  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 01:59:55.308689  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 01:59:55.326167  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:55.344066  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:55.358155  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:55.364372  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:55.382728  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:55.402313  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:55.425025  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:55.566085  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 01:59:55.602183  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:55.605499  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:55.606858  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:55.625486  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 01:59:55.626443  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:55.708098  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:55.726445  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 01:59:55.732378  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 01:59:55.741802  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:55.784559  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:55.803972  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 01:59:55.809556  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 01:59:55.870457  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 01:59:55.922910  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:55.972975  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:55.973047  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:55.974134  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:55.977257  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:56.039003  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:56.059541  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 01:59:56.129729  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:56.140267  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:56.164530  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:56.174046  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:56.176635  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:56.190608  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 01:59:56.192677  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 01:59:56.212649  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:56.259474  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 01:59:56.277586  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 01:59:56.279528  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:56.301501  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:56.464390  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:56.476464  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:56.481438  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:56.501837  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:56.502025  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:56.508072 15868 raft_consensus.cc:482] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 01:59:56.508258 15868 raft_consensus.cc:504] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 01:59:56.508930 15868 leader_election.cc:288] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 01:59:56.512804  6000 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:56.526648  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:56.539438  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:56.554492  6135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 01:59:56.560598  5999 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:56.560889  5999 leader_election.cc:302] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 01:59:56.561262 15868 raft_consensus.cc:2726] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:59:56.639698  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:56.715174  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 01:59:56.752135  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:56.788044  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:56.801769  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:56.842583  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:56.860409  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:56.866863  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 01:59:56.883551  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:56.895287  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 01:59:56.937691  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:56.943151  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:56.964524  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:56.981292  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:56.983399  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:56.988962  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 01:59:57.012321  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:57.049468  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:57.097107  6269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240502 01:59:57.105512  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:57.156512  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 01:59:57.274778  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:57.276826  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:57.322942  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:57.336452  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 01:59:57.352596  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:57.404081  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:57.412175  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:57.431200  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:57.448263  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:57.465962  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 01:59:57.481441  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:57.519891  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:57.545110  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:57.568033  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 01:59:57.571821  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 01:59:57.572072  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:57.577294  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:57.590653  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:57.621405  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 01:59:57.713061  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 01:59:57.729537  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 01:59:57.756419  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:57.778055  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:57.809608  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:57.811244  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:57.877820  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:57.943827  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:58.034971  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:58.036840  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:58.049883  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 01:59:58.050261  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 01:59:58.057799  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:58.088254  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:58.110024  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:58.116521  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:58.127805  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 01:59:58.170917  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:58.216349  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 01:59:58.217490  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 01:59:58.239440  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 01:59:58.247951  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 01:59:58.316396  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:58.316390  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:58.340260  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:58.345732  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:58.383229  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:58.440989  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:58.453361  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:58.484889  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:58.538872  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 01:59:58.621260  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:58.637369  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:58.644883  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 01:59:58.656844  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 01:59:58.686589  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 01:59:58.692166  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:58.713111  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 01:59:58.718250  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:58.728526  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:58.733518  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:58.900594  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:58.906769  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 01:59:58.925628  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 01:59:58.982805  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 01:59:58.986317  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:58.994657  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:59.007963  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 01:59:59.018379  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:59.035948  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:59.062247  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:59.234831  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:59.251281  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:59.280961  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:59.302815  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:59:59.330734  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 01:59:59.335984  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:59.338557  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:59.356693  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 01:59:59.366483  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:59.377612  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 01:59:59.409216  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:59.426755  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:59:59.491910  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:59.560724  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:59:59.562186  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 01:59:59.585829  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 01:59:59.595295  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 01:59:59.613483  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 01:59:59.667449  6402 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 77 similar messages]
I20240502 01:59:59.719411  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:59:59.728499  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:59:59.735571  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:59.737524  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:59:59.742252  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:59:59.752534  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 01:59:59.754022  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:59:59.840622  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 01:59:59.947850  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:00:00.008985  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:00.057490  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:00.071775  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:00.077913  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:00:00.089016  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:00.089035  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:00.091465  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:00.111479  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:00:00.116941  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 02:00:00.118599  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:00.156816  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:00.234869  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 02:00:00.241044  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:00.258133  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:00.279009  5999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 75 similar messages]
I20240502 02:00:00.291137  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:00.293581  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:00.303234  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:00.346168  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:00.475399  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:00:00.504114  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 02:00:00.506016  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 02:00:00.521379 16437 raft_consensus.cc:482] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 02:00:00.521586 16437 raft_consensus.cc:504] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 02:00:00.522387 16437 leader_election.cc:288] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 02:00:00.524556  5999 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:00.525389  6000 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:00.525521  6000 leader_election.cc:302] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 02:00:00.526065 16437 raft_consensus.cc:2726] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 02:00:00.553261  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:00.562886  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:00.564172  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:00.575040  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:00.582624  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:00.590673  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:00.601703  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:00.620311  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:00.830668  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:00.850049  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:00.855875  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:00.864102  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:00:00.865566  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 02:00:00.867161  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:00.888108  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:00.927342  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:01.013284  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:01.108716  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:01.113332  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:01.120434  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:00:01.124985  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:01.131841  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:00:01.199772  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:00:01.203585  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 02:00:01.203975  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:01.215816  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:01.253629  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 02:00:01.278029  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:01.296561  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 02:00:01.299474  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:01.384557  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:01.400318  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:01.412248  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 02:00:01.437036  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:01.442442  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:01.442693  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:01.462265  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:01.469486  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:01.590111  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:01.591965  6136 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240502 02:00:01.614323  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:01.615649  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:01.662832  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:01.708317  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:01.715454  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:01.738431  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:01.808637  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:00:01.829936  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:00:01.841938  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:00:01.892987  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 02:00:01.971582  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:01.999466  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:02.051239  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:02.057137  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:02.061657  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:02.070401  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:02.090853  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:02.092731  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:00:02.109035  6269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240502 02:00:02.118680  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:02.120515  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:02.153605  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:02.207584  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:02.237803  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:02.272810  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:02.301818  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:02.345875  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:00:02.382516  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 02:00:02.393622  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:02.406232  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:02.422684  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:02.435811  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:02.437685  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:02.444161  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:02.464079  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:00:02.594158  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:02.598253  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:02.616658  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:00:02.659819  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:00:02.668218  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:00:02.716195  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:00:02.790628  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 02:00:02.811300  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:02.843849  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:02.856822  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:02.878582  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:02.883083  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 02:00:02.894037  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:02.909646  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:02.942330  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:02.957095  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:02.965286  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:03.014454  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:03.036070  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:03.036523  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:03.138934  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:03.153468  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:03.196656  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:03.214829  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:03.216549  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 02:00:03.241806  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:03.248309  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:03.273424  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:03.281822  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:03.288825  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 02:00:03.448191  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:03.455703  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:03.482787  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 02:00:03.512646  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:03.524283  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 02:00:03.528028  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:03.528784  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:03.533680  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:03.566390  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:00:03.576311  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:03.704564  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:00:03.717610  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:03.721019  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 02:00:03.819795  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:03.850464  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:03.851477  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:03.859100  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:03.868777  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 02:00:03.872478  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:03.911705  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 02:00:03.930063  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:03.960319  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:03.980458  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:03.987684  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:04.005475  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:04.012039  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:04.015583  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:04.048214  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 02:00:04.076352  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:04.230958  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:04.269308  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:04.273561  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 02:00:04.287287  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:04.291551  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:04.295941  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:04.326339  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:04.333679  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:04.397274  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:04.451236  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 02:00:04.470937  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:04.488678  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:04.541589  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:04.546566  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:04.548520  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:04.556115  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:04.561049  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:00:04.577131  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 02:00:04.668720  6402 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 02:00:04.691228  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:04.701573  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 02:00:04.762034  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:04.788100  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:04.804566  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:04.815090  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:04.845325  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 02:00:04.862396  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:04.890028  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:04.894038  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:00:04.980402  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:05.012859  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240502 02:00:05.098269  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 02:00:05.115701  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:05.116169  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:05.123068  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:05.123392  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:05.125360  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:05.154385  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 02:00:05.166934  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:05.205403  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:05.207494  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 02:00:05.257288  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:05.278009  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:05.290990  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:05.316535  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:05.333166  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:05.386873  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:05.429471  6000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 02:00:05.432214  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:05.435647  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:00:05.459219  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:00:05.575054  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 02:00:05.636898  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:05.654356  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:05.658671  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 02:00:05.663069  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:05.666316  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:05.711742  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:05.769332  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:05.788561  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:05.863224  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 02:00:05.938110  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:05.959393  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:05.961697  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:05.964700  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:06.000101  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 02:00:06.023372  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:06.033329  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:06.040530  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:06.147578  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:00:06.204535  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:06.260391  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:06.285722  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:06.321322  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:06.325342  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:00:06.330513  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 02:00:06.331351  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:06.376245  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:06.384049  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:06.393085  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:00:06.403535  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:00:06.409389  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:00:06.414080  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:00:06.545854  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:06.546169  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:06.547713  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:06.550827  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:06.574445  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:06.578480  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:06.589154  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:06.593576  6135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 02:00:06.604004  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:00:06.609269  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:06.715708  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:06.753037  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:06.761976  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:06.790516  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:06.826256  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 02:00:06.845914  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:06.859230  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:07.001936  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:07.048364  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:07.054759  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:07.060757  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:07.086513  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:07.091120  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:07.151767  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:07.165313 17400 raft_consensus.cc:482] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 02:00:07.165535 17400 raft_consensus.cc:504] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 02:00:07.166304 17400 leader_election.cc:288] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 02:00:07.167068  6269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240502 02:00:07.168565  6270 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:07.171969  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:07.180342  6269 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:07.180527  6269 leader_election.cc:302] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 02:00:07.180819 17400 raft_consensus.cc:2726] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:00:07.277769  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 02:00:07.311112  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:07.338754  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:07.352114  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 02:00:07.353482  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:07.374420  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:07.386530  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 02:00:07.387349  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:07.406960  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:07.436180  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:07.461500  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:00:07.543545  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 02:00:07.590910  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:07.599220  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 02:00:07.607070  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:07.610464  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:07.611688  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:07.631583  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:00:07.659695  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 02:00:07.665697  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:07.687238  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:07.751478  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:07.879263  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:07.897912  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:07.917584  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:00:07.930120  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 02:00:07.931942  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:07.955526  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 02:00:07.959343  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:07.994496  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:08.001706  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:08.020627  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:08.099117  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:00:08.127074  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 02:00:08.141003  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:08.150370  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 02:00:08.170329  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:08.182471  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:08.210415  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:08.212335  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:08.227761  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:08.241305  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:08.260466  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:08.346246  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:08.361486  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:08.373049  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:08.407541  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:08.414341  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:08.454119  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:08.568840 16741 raft_consensus.cc:482] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 02:00:08.569103 16741 raft_consensus.cc:504] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 02:00:08.569964 16741 leader_election.cc:288] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 02:00:08.577384  5999 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:08.584612  6000 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:08.584909  6000 leader_election.cc:302] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 02:00:08.585561 16741 raft_consensus.cc:2726] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:00:08.618959  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:00:08.635937  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:00:08.663405  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:08.723647  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:00:08.827258  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:00:08.828473  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 02:00:08.833680  6447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:08.833990  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:08.854465  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:00:08.875988  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:08.923606  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:08.936337  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:08.950734  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 02:00:08.952306  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:08.980026  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:08.985934  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:09.009012  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:09.010957  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 02:00:09.011482  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:09.054499  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:09.120465  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:09.125784  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:09.153280  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:09.170264  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:09.177965  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:09.220562  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:09.246855  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:00:09.281814  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:09.290091  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:09.290138  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:09.400563  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:09.446810  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 02:00:09.495419  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:09.513231  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:09.527562  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:09.542510  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:09.557729  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:09.645684  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:09.703717  6401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240502 02:00:09.704985  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 02:00:09.714761  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 02:00:09.717989  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:00:09.816533  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:00:09.838583  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:09.841650  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:09.846464  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:09.904220  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:09.957949  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:09.970027  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:09.990904  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:10.017572  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:10.096254  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 02:00:10.100735  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:10.151360  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:10.152838  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:10.164223  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:10.168512  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:10.173118  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:10.177361  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 02:00:10.219023  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:10.243108  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:10.248479  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:10.269351  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:10.285213  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 02:00:10.308211  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:10.336990  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:10.362651  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:10.365031  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:00:10.394918  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:00:10.398344  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:10.468358  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 02:00:10.524636  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:10.536139  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:10.538069  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:10.540904  5999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 02:00:10.544808  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 02:00:10.554311  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:10.567725  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:10.586403  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 02:00:10.594753  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:10.778153  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:00:10.803047  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:10.814142  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 02:00:10.881816  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:10.888885  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:10.903302  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:10.908211  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:10.915786  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:10.949419  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:10.968894  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:10.971984  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:11.025701  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:11.036435  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:11.040920  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:11.105508  6134 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.135:40543, user_credentials={real_user=slave}} blocked reactor thread for 70716.5us
I20240502 02:00:11.121817  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:11.133838  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:11.158294  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:00:11.221622  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:11.277992  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:00:11.288707  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:11.339753  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:00:11.340050  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 02:00:11.383157  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:11.428956  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 02:00:11.430225  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:11.441987  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:11.453281  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 02:00:11.472020  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:11.496376  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:11.504948  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:11.511526  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:11.525930  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:11.565745  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:11.603399  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:11.611446  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:11.654630  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 02:00:11.673831  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:11.674201  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:00:11.729632  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:00:11.745292  6136 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 02:00:11.746299  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:11.770804  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:11.884676  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 02:00:11.906683  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:11.907531  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:11.931010  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:11.946650  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:11.955650  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:11.964692  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:11.994884  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:00:12.051683  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:00:12.123667  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 02:00:12.148473  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:12.164534  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 02:00:12.164912  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:12.168331  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:12.172525  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:12.221401  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:12.221876  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:12.259370  6269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 02:00:12.276811  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:12.357851  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:12.416206  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:12.422580  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:12.429970  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:12.449205  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:12.482838  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:12.531239  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:12.686968  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 02:00:12.755070  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:12.764055  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:12.767341  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:12.774700  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:12.786425  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:12.787009  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:12.794327  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:12.816129  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 02:00:12.881453  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:12.913866  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:12.934979  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:12.935266  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 02:00:12.954670  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:12.968983  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:13.005436  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 02:00:13.009480  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:13.037233  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:13.041988  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:00:13.067945  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:00:13.069263  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:00:13.080745  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:00:13.137282  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:00:13.211344  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:13.216063  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:13.217690  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:13.232234  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:13.246816  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:13.254674  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:13.273569  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:13.354496  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 02:00:13.425467  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:13.447968  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:13.450649  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:13.451524  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:13.465263  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:13.486469  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:13.509169  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:13.592329  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:00:13.657151  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:13.673272  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 02:00:13.703605  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:13.711959  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:13.732568  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:13.739511  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:13.767341  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:13.794533  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 02:00:13.796221  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:13.840142  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:00:13.924602  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:13.977500  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 02:00:13.986518  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:13.991904  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:14.003145  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:14.006489  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:14.024296  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:14.038429  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:14.064438  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:00:14.112181  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 02:00:14.185945  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:14.186513  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:14.197531  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:14.199440  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:14.202819  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 02:00:14.217574  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:14.245378  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:14.252903  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:00:14.281056  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:14.318807  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:00:14.364585  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:00:14.365949  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:00:14.407943  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:14.520100  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 02:00:14.566154  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:14.582443  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:14.607692  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:14.626741  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:14.636488  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:14.652943  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:14.690371  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:14.704568  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:14.707263  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:14.716868  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:14.744732  6401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 77 similar messages]
W20240502 02:00:14.756953  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:00:14.807554  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:14.819293  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:14.827342  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:14.831501  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:14.860076  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:14.913587  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:14.922979  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:14.935514  8249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:14.940904  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:14.970808  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:15.229079  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:15.232035  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:15.261572  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:15.300449  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:15.310613  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:15.321511  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 02:00:15.334208  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:15.389462  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:15.427675  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:00:15.428889  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:15.484318  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:00:15.501773  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:00:15.511685  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:00:15.522125  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 02:00:15.543360  5999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 71 similar messages]
I20240502 02:00:15.557341  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:15.557760  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:15.593142  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:15.616076  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:15.619827  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:15.649825  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 02:00:15.677505  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:15.759645  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 02:00:15.821462  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:15.842429  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:15.858402  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:00:15.859170  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 02:00:15.874671  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:15.887701  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:15.893479  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:15.912199  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:15.929831  5998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.135:40543, user_credentials={real_user=slave}} blocked reactor thread for 69975us
I20240502 02:00:15.963704  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:15.963814  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:15.964946  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:15.997555  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:16.019323  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:16.032866  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 02:00:16.159427  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:16.173537  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:16.194829  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 02:00:16.209589  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:16.295542  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:16.354128  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 02:00:16.391974  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:16.420094  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:16.455662  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:16.455816  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:16.458494  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:16.515329  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:16.523674  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:16.573794  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:00:16.603022  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:00:16.605108  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:00:16.626775  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:00:16.665745  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 02:00:16.680328  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:16.699443  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 02:00:16.711472  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:16.737468  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:16.779309  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:16.786484  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:16.813444  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 02:00:16.849588  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:16.865391  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:00:16.904484  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:16.929320  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:00:16.942082  6136 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:00:16.943260  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:17.012776  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:17.025260  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:17.038386  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:17.056913  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:17.060570  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 02:00:17.072170  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:17.084097  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:17.134963  6403 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.130:38197, user_credentials={real_user=slave}} blocked reactor thread for 55137.5us
W20240502 02:00:17.199524  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 02:00:17.274938  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:17.285693  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:17.289584  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:17.310271  6270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240502 02:00:17.318085  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:17.318488  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:17.320796  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:17.439296  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:17.457401  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:17.541239  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:17.550529  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:17.561309  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:17.606285  6000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.131:44245, user_credentials={real_user=slave}} blocked reactor thread for 47114.2us
I20240502 02:00:17.617354  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:17.658741  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:17.682176  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:17.727998  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:17.839269  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:17.843427  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:17.875988  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:17.876467  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 02:00:17.884274  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:17.918044  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:17.972819  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 02:00:18.002435  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:18.022884  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 02:00:18.046566  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:18.087015  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:00:18.138159  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 02:00:18.178202  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:18.181098  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:00:18.184039  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:18.211283  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:18.220319  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 02:00:18.221236  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:18.242983  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:18.255844  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:18.268355  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:18.273013  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 02:00:18.292948  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:18.324995  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:18.336584  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:18.368773  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:18.438866  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:18.446094  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 02:00:18.496716  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:18.498476  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:18.501163  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:00:18.609699  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:00:18.698765  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:00:18.704484  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:00:18.749006  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:18.759649  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 02:00:18.842662  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:18.862468  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:18.883553  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:18.894419  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:18.896363  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:18.902607  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:18.908347  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:19.037972  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 02:00:19.047009  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:19.077709  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:19.113543  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:19.130510  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 02:00:19.132434  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:19.151465  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:19.176621  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:00:19.182698  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 02:00:19.183698  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:19.203526  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:19.248999  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:00:19.252539  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:00:19.292644  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 02:00:19.335716  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:19.355105  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 02:00:19.361502  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:19.412187  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:19.424921  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:19.442595  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:19.471619  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:19.485093  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:19.504622  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 02:00:19.619230  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:19.634984  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:19.653077  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:19.655434  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:19.687520  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:19.687664  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:19.693257  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 02:00:19.751955  6401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 02:00:19.753111  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:19.851120  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:19.852015  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:19.859179  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:19.886916  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:19.898563  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:19.952083  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:19.968678  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:20.193902  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:20.231971  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:20.242549  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:20.255352  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:20.256429  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:20.290957  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:20.304239  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:20.324788  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:20.332603  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:20.339231  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:20.342837  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 02:00:20.348781  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:20.380805  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:20.402215  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:20.430912  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:00:20.482260  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:00:20.503358  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:20.503387  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:00:20.567401  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 02:00:20.619917  5999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240502 02:00:20.667455  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:20.675700  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:20.679735  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:20.694422  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:20.702648  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:20.702857  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:20.706670  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:20.751766  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:00:20.845947  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:00:20.931063  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:00:20.959846  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:00:20.992899  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 02:00:21.002528  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:21.021514  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:21.029007  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 02:00:21.030530  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:21.042133  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:21.048439  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:21.068939  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:21.074234  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:21.079213  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:21.079671  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:21.096284  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:21.114226  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 02:00:21.122749  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:21.125321  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:21.140527  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:21.168102  6134 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.135:40543, user_credentials={real_user=slave}} blocked reactor thread for 46425.5us
W20240502 02:00:21.173398  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 02:00:21.178962  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:21.388726  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:21.412873  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:21.418294  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:21.445936  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:21.447984  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:21.477561  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:21.492600  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:21.497653  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:00:21.546720  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:00:21.585521  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:21.629264  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:00:21.673736  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:00:21.675115  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 02:00:21.681556  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:21.709017  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:21.709151  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:21.741192  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:21.742200  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:21.770735  6134 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.132:36867, user_credentials={real_user=slave}} blocked reactor thread for 73607us
W20240502 02:00:21.780784  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:00:21.790844  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 02:00:21.817378  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:21.908592  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:00:21.940222  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:00:21.943544  6136 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:00:21.945290  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:21.985672  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:00:22.022943  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 02:00:22.052181  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:22.065730  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:22.093796  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:22.118461  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:22.122021  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:22.142943  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:22.183140  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:22.217294  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:22.241911  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:22.290289  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:22.335609  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:22.339571  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:22.343487  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:22.370079  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:22.444944  6270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 02:00:22.501935  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:22.512809  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:22.523329  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:22.528668  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:22.559301  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:22.570663  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:22.596825  6403 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.130:38197, user_credentials={real_user=slave}} blocked reactor thread for 67488.5us
I20240502 02:00:22.630129  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:22.768693  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:22.887239 18629 raft_consensus.cc:482] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 02:00:22.887459 18629 raft_consensus.cc:504] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 02:00:22.888315 18629 leader_election.cc:288] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 02:00:22.892663  5999 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:22.953227  6000 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:22.953542  6000 leader_election.cc:302] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 02:00:22.953861 18629 raft_consensus.cc:2726] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:00:22.977861  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:22.985321  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:00:23.004141  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 02:00:23.006053  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:23.030465  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 02:00:23.049454  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:23.061019 19326 raft_consensus.cc:482] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 02:00:23.061249 19326 raft_consensus.cc:504] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 02:00:23.062047 19326 leader_election.cc:288] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 02:00:23.066588  6269 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:23.068917  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:23.070648  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:23.101119  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 02:00:23.119728  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:23.120800  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:23.128712  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:23.137094  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:23.158658  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 02:00:23.167734  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 02:00:23.207010  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:23.228562  6270 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:23.229856  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:23.231590  6270 leader_election.cc:302] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 02:00:23.244493  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:23.250488  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:23.251891  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:23.284142  6268 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.135:40543, user_credentials={real_user=slave}} blocked reactor thread for 62188us
I20240502 02:00:23.291308  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:23.305233 19844 raft_consensus.cc:2726] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 02:00:23.312815  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:23.315285  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:23.315642  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:23.358575  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:23.372924  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:23.390319  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:00:23.409168  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:00:23.409296  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:00:23.446273  6400 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.135:40543, user_credentials={real_user=slave}} blocked reactor thread for 58274.5us
W20240502 02:00:23.462920  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:00:23.513315  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:23.589547  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:00:23.600692  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 02:00:23.669168  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:23.681666  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:23.684551  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:23.700891  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:23.704469  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:23.728399  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:23.845310  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:00:23.856266  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:00:23.891022  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:23.931264  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:00:23.946027  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:00:23.967242  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:24.062510  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:24.076287  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 02:00:24.077559  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:24.101081  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:24.125324  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:24.125761  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:24.135120  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:00:24.153394  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 02:00:24.164330  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:24.184016  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:24.190084  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 02:00:24.219033  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:24.238127  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:24.244385  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:24.252528  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:24.287904  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:24.375159  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:24.443342  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:24.492326  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:00:24.503126  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 02:00:24.503358  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:24.513373  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 02:00:24.513548  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:24.520454  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:24.547351  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:24.558162  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:24.558625  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:24.609422  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 02:00:24.924983  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:24.934418  6401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 82 similar messages]
W20240502 02:00:24.969327  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:24.971300  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:24.980129  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:24.985365  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:24.989959  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:25.051631  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:25.074497  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:25.088551  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:25.133852  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:25.149001  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:25.183390  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:25.216940  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:25.242468  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:25.328984  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:25.332329  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:25.347270  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:25.350569  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:25.369225  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:25.380973  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:25.399507  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:25.496230  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:00:25.501027  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:25.549293  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:00:25.612545  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:00:25.623600  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 02:00:25.665809  5999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 02:00:25.677909  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 02:00:25.774328  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:25.811997  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 02:00:25.815057  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:25.825165  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:00:25.834547  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:25.837769  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:25.838292  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:25.844992  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:25.853507  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 02:00:25.857942  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:25.878541  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:25.939468  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 02:00:25.944743  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:25.957607  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:25.961943  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:25.996356  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:26.005115  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:26.028977  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 02:00:26.040143  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:26.064258  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 02:00:26.136620  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:26.138608  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:26.159911  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:26.201772  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:26.208760  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 02:00:26.216316  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:26.219240  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:26.269083  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:00:26.276434  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:00:26.358296  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:26.414073  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:26.415271  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:26.417237  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:26.425767  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:26.434779  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:26.445809  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:26.449512  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:26.466377  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:00:26.498522  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:00:26.650890  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:00:26.702371  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:00:26.707185  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 02:00:26.795737  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:26.833895  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:26.853729  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:26.860481  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:26.868597  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:26.875102  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:26.901237  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:26.954668  6135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 02:00:26.962785  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 02:00:26.965468  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:26.967217  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:26.969136  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:26.990037  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:27.014653  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:27.020102  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:27.024132  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 02:00:27.074391  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:27.083578  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 02:00:27.096148  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:27.106016  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 02:00:27.114078  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:27.156872  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:27.224414  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:27.276109  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:27.283056  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:27.352483  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:00:27.460041  6269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 02:00:27.474891  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:27.556319  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:27.622454  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:27.744944  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:27.774223  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:27.779855  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:27.809319  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:27.839159  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:27.926680  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:27.990962  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:28.006418  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:28.025223  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:00:28.026923  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:00:28.035117  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:28.040783  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:28.061949  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:28.063084  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:28.074627  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 02:00:28.106004  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:28.158748  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:00:28.184944  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 02:00:28.230551  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:28.238899  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:28.254153  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:28.261993  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:28.268407  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:28.270627  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:28.346232  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:00:28.406117  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:00:28.460171  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:28.470844  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:00:28.532238  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:00:28.538843  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:00:28.596678  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:00:28.610042  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:28.628026  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:28.642432  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:28.642500  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:28.660571  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 02:00:28.673563  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:28.746253  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:28.783020  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 02:00:28.792789  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:28.800464  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:28.853365  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:28.872551  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:28.873028  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:28.895854  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:28.897341  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 02:00:28.932837  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:28.955830  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 02:00:28.980675  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:29.002983  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:29.010973  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 02:00:29.017910  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:29.036165  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:29.048804  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:29.054834  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:29.069150  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:29.079953  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 02:00:29.093545  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:29.196832  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:00:29.207641  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:00:29.285740  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:00:29.296702  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:29.305351  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:29.305389  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:29.315964  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:29.321241  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:29.341754  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:29.371316  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:29.376881  5998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.135:40543, user_credentials={real_user=slave}} blocked reactor thread for 49097us
W20240502 02:00:29.505656  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:29.540416  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:00:29.602298  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 02:00:29.651409  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:00:29.708680  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 02:00:29.722659  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:29.757596  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:29.794469  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:29.805514  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:29.842741  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:29.858740  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:29.883105  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:29.889966  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:30.054730  6401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 83 similar messages]
I20240502 02:00:30.069016  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:30.072341  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:30.074868  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:30.092499  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:30.147034  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:30.202090  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:30.251211  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:30.278100  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:30.278286  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:30.278791  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:30.283494  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:30.284618  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:30.326177  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:30.357509  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:00:30.463245  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:30.544349  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:00:30.572582  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:00:30.572559  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 02:00:30.620632  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:30.650532  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:30.653565  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:30.674888  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:30.676810  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:30.700030  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:00:30.723461  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 02:00:30.724347  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:30.735884  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:30.738953  5999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 02:00:30.740608  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:30.742826  6270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.133:43379, user_credentials={real_user=slave}} blocked reactor thread for 112754us
I20240502 02:00:30.759274  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:30.766469  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:30.796020  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:30.869218  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 02:00:30.872189  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:30.875131  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:30.958477  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:31.028788  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:31.034361  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:31.043069  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:31.046742  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:31.057955  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:31.062141  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:31.076380  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:00:31.082365  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:00:31.162823  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:00:31.183796  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:00:31.239068  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:00:31.316246  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:31.366494  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:31.384105  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 02:00:31.391381  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:31.402416  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:31.412176  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:31.438720  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:31.441551  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:31.445255  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:31.492187  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 02:00:31.595136  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:31.596200  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:31.597033  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:00:31.621929  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:00:31.639642  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 02:00:31.642199  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:31.662379  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:00:31.669622  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 02:00:31.692667  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:31.723402  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:31.745167  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:31.763803  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:00:31.767010  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:31.915180  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:31.933188  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:31.971241  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:31.988116  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:31.991436  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:31.991402  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:32.008474  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 02:00:32.023366  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:00:32.033721  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:32.132079  6268 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.135:40543, user_credentials={real_user=slave}} blocked reactor thread for 90198.2us
W20240502 02:00:32.140183  6135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:00:32.147367  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:00:32.158216  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 02:00:32.189091  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:32.225507  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:32.227133  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:32.228264  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:32.367630  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:32.378942  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:32.394426  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:32.461206  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:32.462677  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:32.485399  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:32.490255  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:32.490904  6269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240502 02:00:32.503152  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:32.531168  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:32.647581  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:32.844528  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:32.854997  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:32.870659  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:32.890623  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:32.920728  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:32.926429  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:32.949458  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:33.062146  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 02:00:33.086374  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:33.093510  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:33.129861  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:33.130573  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:33.203109  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:33.210897  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:00:33.211699  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:00:33.211697  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 02:00:33.225787  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:33.232074  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:33.293236  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:00:33.332666  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 02:00:33.350862  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:33.371809  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:33.374892  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:33.392872  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 02:00:33.402737  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:33.419397  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:33.468287  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:33.486181  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:33.586051  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:00:33.617523  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:00:33.633180  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:00:33.669090  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:33.688153  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:00:33.729566  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:00:33.729563  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 02:00:33.739161  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:33.759847  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:33.772709  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:33.796197  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:33.799295  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 02:00:33.820257  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:33.820351  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:33.837010  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:33.859236  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:33.886866  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:33.899484  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:33.929692  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:33.947382  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:33.989390  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:34.137859  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:00:34.163399  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:00:34.199235  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 02:00:34.229774  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:34.233069  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:34.236886  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:34.251052  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:34.259831  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:34.261235  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:34.300748  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 02:00:34.313619  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:34.345552  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:00:34.430145  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:00:34.488792  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 02:00:34.509677  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:34.514223  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:34.516685  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:34.536429  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:34.538843  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:34.560860  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:34.561991  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:34.577121  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:00:34.623073  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 02:00:34.644848  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 02:00:34.795737  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:34.813426  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:34.819736  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:34.853679  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:34.917476  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:34.930610  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:34.963587  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:35.092609  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:35.095212  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:35.104269  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:35.106554  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:35.118499  6401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 78 similar messages]
I20240502 02:00:35.127269  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:35.148545  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:35.160874  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:35.315986  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:35.363344  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:35.371054  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 02:00:35.385360  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:35.407294  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:35.410246  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:35.426280  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:35.447152  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:35.571529  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:00:35.666714  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:35.682318  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:35.721624  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:35.761847  6000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 02:00:35.762537  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 02:00:35.764621  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:35.780992  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:35.797914  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 02:00:35.803364  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:35.838486  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:35.840951  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:35.875036  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:00:35.937932  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 02:00:35.983564  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:36.006603  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:36.010371  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:36.029310  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 02:00:36.030546  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:36.059655  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 02:00:36.064023  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:36.107334  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:36.118579  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:36.133020  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:00:36.137873  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:36.143216  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:36.143774  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:00:36.147994  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 02:00:36.169893  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:36.233909  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:36.320351  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:36.322942  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:36.379544  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:36.394802  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:00:36.446414  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:00:36.457680  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:00:36.485723  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:00:36.505702  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 02:00:36.511001  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:36.522462  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:36.524199  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:36.525295  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:36.528584  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:36.558372  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:36.577525  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:36.587512  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:00:36.696238  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 02:00:36.715162  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:36.725415  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:36.798566  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:36.825330  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:36.849287  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 02:00:36.850172  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:36.850945  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:36.869531  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:00:36.873818  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:00:36.997820  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:37.021042  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:37.024621  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:37.054771  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:37.065693  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:37.072283  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:37.077781  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:37.139874  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:37.200644  6135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 02:00:37.202536  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 02:00:37.243360  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 02:00:37.273252  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:37.302367  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:37.338990  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:37.343405  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:37.354049  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:37.356149  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:37.371255  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 02:00:37.414132  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:37.519824  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:37.536988  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:37.554931  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:37.557829  6270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240502 02:00:37.558714  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:37.600013  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:37.638556  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:37.639642  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:37.846261  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:37.847107  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:37.850097  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:37.850939  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:37.875367  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:37.916149  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:37.919374  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 02:00:37.929853  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:38.028273  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:38.098526  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:38.123852  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:38.130491  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:38.134303  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:38.148494  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:38.153956  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 02:00:38.167913  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:38.205155  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:00:38.281906  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 02:00:38.328827  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:38.336098  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:38.339959  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 02:00:38.365175  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:38.372895  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:38.374320  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:38.375540  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:38.412993  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:38.433445  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:38.471191  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:00:38.510628  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 02:00:38.568626  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:38.578564  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:38.587950  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:38.597020  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 02:00:38.624626  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:38.652427  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:38.672971  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:38.678179 21740 raft_consensus.cc:482] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 02:00:38.678357 21740 raft_consensus.cc:504] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 02:00:38.679106 21740 leader_election.cc:288] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 02:00:38.681461  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:00:38.682861  5999 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:38.683148  6000 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:38.683241  6000 leader_election.cc:302] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 02:00:38.683763 21740 raft_consensus.cc:2726] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:00:38.700740  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:00:38.764294  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:00:38.803455  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:00:38.858184  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 02:00:38.904846  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:38.927560  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:38.930860  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:38.932927  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:00:38.945883  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 02:00:38.975294  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:38.976419  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:38.988440  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:38.991127  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:39.017453  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:39.053519 21115 raft_consensus.cc:482] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 02:00:39.053711 21115 raft_consensus.cc:504] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 02:00:39.054433 21115 leader_election.cc:288] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 02:00:39.057566  6269 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:39.059830  6270 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:39.059973  6270 leader_election.cc:302] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
W20240502 02:00:39.060461  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 02:00:39.060720 21115 raft_consensus.cc:2726] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 02:00:39.191112  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:39.204195  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:39.243229  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:39.248999  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:39.269227  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:39.296028  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:39.319329  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 02:00:39.350442  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:39.386773  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 02:00:39.397136  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:39.415050  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 02:00:39.436167  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:39.453188  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 02:00:39.454213  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:39.456324  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 02:00:39.464880  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:39.474001  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:39.538192  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:39.606307  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:39.661645  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 02:00:39.800431  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:39.820745  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:39.851152  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 02:00:39.852006  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:39.864185  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:39.930936  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:39.933771  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:39.956297  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:39.973309  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:39.977421  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 02:00:39.984344  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:39.985463  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:39.990547  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:40.018002  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:40.025993  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:40.121191  6402 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 78 similar messages]
I20240502 02:00:40.182669  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:40.188284  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:40.209728  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:40.216568  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:40.224576  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:40.229247  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:40.245338  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:40.263760  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 02:00:40.422636  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:40.429826  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:40.450439  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:40.455214  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:40.456542  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:40.471825  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:40.567425  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:40.616370  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:40.726056  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:40.740759  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:40.753613  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:40.803946  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 02:00:40.834826  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:40.839774  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:00:40.835753  6000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 02:00:40.861526  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 02:00:40.877717  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:40.904763  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:40.927103  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:00:40.990821  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:00:40.997840  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:00:41.012071  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:41.031227  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:41.034443  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:41.053090  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 02:00:41.075759  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:41.079084  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:41.093111  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:41.105334  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:41.108758  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:00:41.145689  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:00:41.160109  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:41.186081  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 02:00:41.235929  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:41.236949  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:41.241362  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:41.243738  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:41.279968  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:41.289377  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:41.322805  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:00:41.338701  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:00:41.413444  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 02:00:41.511262  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:41.513720  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:41.522053  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:41.527606  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:41.547142  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:41.553522  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:41.577761  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:41.590601  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:41.660013  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:00:41.721071  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 02:00:41.729530  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:41.760567  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:41.780324  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:41.831676  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:41.874572  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:00:41.882805  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 02:00:41.907733  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:41.918531  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:41.943362  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:00:42.021834  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:42.022126  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 02:00:42.028239  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:42.034243  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:42.040465  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:42.065882  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:42.087180  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:42.100071  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:42.102694  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:42.104096  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:00:42.122345  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:00:42.126616  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 02:00:42.209359  6135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 02:00:42.266537  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 02:00:42.452903  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:42.478463  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:42.505348  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:42.515450  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:42.524039  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:42.530874  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:42.558521  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:42.585942  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:42.587342  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:42.588285  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:42.604043  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:42.616813  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:42.623608  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:42.645089  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:42.670382  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:42.673184  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:42.677363  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:42.720367  6270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240502 02:00:42.738379  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:42.748275  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:42.784091  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:42.784546  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 02:00:43.047025  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:43.048151  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:43.072135  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:43.073310  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:43.075906  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:43.091526  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:43.105705  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:43.120640  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:00:43.183835  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:43.456630  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:43.473858  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:43.500556  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:43.508553  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:00:43.528370  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:00:43.530741  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 02:00:43.535528  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:43.537959  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:43.539480  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:00:43.557876  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:00:43.562193  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 02:00:43.566475  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:43.574347  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:43.580224  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 02:00:43.600132  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:43.611558  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:00:43.635143  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:43.654511  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:43.661037  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:43.661563  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:00:43.664635  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:43.693883  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:43.710995  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:43.711480  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:43.730104  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 02:00:43.752041  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:43.773622  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:43.786129  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:43.792320  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:43.857293  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:43.889050  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 02:00:43.907258  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:00:44.100630  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:44.110247  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:44.110323  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:44.121748  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:44.123663  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:44.125870  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:44.137583  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:44.221680  6271 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.130:38197, user_credentials={real_user=slave}} blocked reactor thread for 47879.4us
W20240502 02:00:44.291316  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:44.303848  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:00:44.321969  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:00:44.329317  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:00:44.412276  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:00:44.424850  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 02:00:44.484073  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:44.496865  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 02:00:44.508291  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:44.511988  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:44.530527  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:44.553406  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 02:00:44.609432  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:44.641846  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:44.644268  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:44.665473  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:44.687656  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:44.712904  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:44.718808  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:44.731297  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:44.777601  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:44.779300  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:44.796608  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:00:44.843572  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:00:44.913123  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 02:00:44.955745  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:44.961551  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:44.961939  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:44.978983  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:44.980535  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:45.002956  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:45.203656  6401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240502 02:00:45.205060  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:45.295253  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:45.327351  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 02:00:45.361651  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:45.383795  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:45.385859  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:45.469347  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:45.494144  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:45.498277  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:45.583081  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:45.601390  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:45.626988  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:45.627427  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:45.655753  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:45.663887  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:45.698320  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:45.722213  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 02:00:45.735441  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:45.758657  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:45.803301  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 02:00:45.818652  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:45.818790  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:45.835886  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:45.845695  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:45.908455  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:45.935119  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:00:45.940011  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 02:00:45.987318 22649 raft_consensus.cc:482] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 02:00:45.987540 22649 raft_consensus.cc:504] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 02:00:45.988382 22649 leader_election.cc:288] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 02:00:45.989228  5999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 02:00:45.990259  5999 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:45.991958  6000 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:45.992089  6000 leader_election.cc:302] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 02:00:45.992381 22649 raft_consensus.cc:2726] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:00:46.016741  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 02:00:46.021131  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:00:46.075973  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:46.107291  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 02:00:46.107820  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:46.108814  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:46.109009  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:46.124366  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:46.131230  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:46.142452  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:46.194808  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:00:46.235960  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:00:46.251845  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:00:46.284242  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:00:46.335464  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:46.345153  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:00:46.484004  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 02:00:46.556671  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:46.563428  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:46.594687  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:46.611624  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 02:00:46.613029  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:46.619424  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:46.621960  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:46.653195  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:46.656404  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:46.671190  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 02:00:46.684512  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:46.693681  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:46.711953  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:46.719079  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:46.745688  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:46.769943  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:00:46.778239  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 02:00:46.806098  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:46.889436  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:46.912741  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:46.946188  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:46.986570  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:46.991022  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:46.994200  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:47.040354  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:00:47.090167  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:00:47.110455  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:47.133479  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:00:47.248306  6135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 56 similar messages]
I20240502 02:00:47.249586  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:47.251260  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:47.262019  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:47.274807  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:47.306456  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:47.316550  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:47.324012  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 02:00:47.336768  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:47.378048  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 02:00:47.539505  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:47.540601  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:47.550482  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 02:00:47.569020  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:47.593952  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:47.600175  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:47.620816  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:47.626411  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:47.740931  6269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 02:00:47.768743  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 02:00:47.802294  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:47.814082  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:47.817934  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:47.841252  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:47.856633  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:47.861719  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:47.886337  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:48.036928  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:48.075251  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 02:00:48.085513  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:48.090050  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:48.116087  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:48.131992  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:48.133114  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:48.133544  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:48.172253  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:48.302326  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 02:00:48.316921  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 02:00:48.473556  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:48.487629  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:48.498155  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:48.514536  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:48.518885  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:48.533846  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:48.544160  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:48.575192  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:48.595266  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 02:00:48.618471  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:48.646471  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:00:48.646986  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 02:00:48.647104  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 02:00:48.679252  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:48.703161  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:48.762251  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:48.766153  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:48.768729  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 02:00:48.818881  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:00:48.819242  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:48.837648  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:00:48.862969  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 02:00:48.906594  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:48.906625  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:48.907828  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:48.935330  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:48.950683  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:48.961524  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 02:00:48.993913  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:49.026824  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 02:00:49.065243  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 02:00:49.065526  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:49.186884  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:49.199025  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 02:00:49.203986  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:49.218775  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:49.224736  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:49.235224  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:49.249390  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:49.262221  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 02:00:49.330569  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:49.342355  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:00:49.463276  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:49.482036  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 02:00:49.508356  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:49.515681  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 02:00:49.518904  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:49.533389  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:49.555313  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:49.558831  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:49.584991  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 02:00:49.610360  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:49.756461  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 02:00:49.771003  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:49.772425  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:49.787761  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:49.813738  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:49.837965  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:49.846287  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:49.852442  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:49.866505  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:49.892210  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:49.923820  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:49.930032  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:49.959307  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:49.960717  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:50.020222  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 02:00:50.037503  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:50.153381  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 02:00:50.200073  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 02:00:50.206403  6401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 84 similar messages]
I20240502 02:00:50.209082  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:50.209628  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:50.232789  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:50.241577  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:50.247371  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:50.277263  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:50.370193  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:50.371775  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:50.384300  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:50.461688  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:50.464411  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:50.491708  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:50.503028  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 02:00:50.503350  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 02:00:50.508029  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:50.524583  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:50.678617  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:50.681375  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:50.700819  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:50.702878  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:50.709806  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:50.710018  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:50.732877  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:00:50.829748  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 02:00:50.863251  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:50.918085  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 02:00:50.928751  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 02:00:50.940522  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:51.005484  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:51.020263  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:51.023097  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:51.044597  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:51.047148  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:51.072086  5999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 72 similar messages]
I20240502 02:00:51.101070  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:51.121219  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:51.149889  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:51.160756  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:51.224334  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:51.243567  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 02:00:51.244735  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:51.253526  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:51.283052  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:51.385929  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:51.426455  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:51.426544  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 02:00:51.439074  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:51.440311  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 02:00:51.450595  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:51.452155  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:51.458400  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:51.464286  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 02:00:51.470285  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:51.478942  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 02:00:51.540872  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 02:00:51.564201  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:00:51.591531  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 02:00:51.674852  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:51.689296  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:51.707299  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 02:00:51.709206  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 02:00:51.711030  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:51.714484  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:51.718351  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:51.761703  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:51.776181  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:51.854801  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:51.920650  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:00:51.925503  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 02:00:51.943722  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:51.981563  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:52.011458  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:52.030470  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 02:00:52.048493  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:52.051504  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:52.059535  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:52.069617  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 02:00:52.271000  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:52.278717  6135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240502 02:00:52.285553  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:52.299970  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:52.300958  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:52.330472  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:52.356815  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:52.357594  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:52.404348  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:52.435706  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:52.450057  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 02:00:52.455071  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 02:00:52.458977  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:52.481400  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:52.548288  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:52.554821  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:52.560328  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:52.587186  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:00:52.594574  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 02:00:52.757558 23688 raft_consensus.cc:482] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 02:00:52.757761 23688 raft_consensus.cc:504] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 02:00:52.758554 23688 leader_election.cc:288] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 02:00:52.761770  6000 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:52.796914  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:52.797796  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:52.799588  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:52.815033  6269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240502 02:00:52.820659  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:52.822403  5999 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:52.822829  5999 leader_election.cc:302] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 02:00:52.823177 23688 raft_consensus.cc:2726] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:00:52.827672  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:52.857600  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:52.863041  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:52.873292  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:52.919616  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 02:00:52.992714  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 02:00:53.059700  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:53.072494  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:53.079097  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:53.100690  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:53.101061  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:53.135349  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:53.162513  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 02:00:53.164633  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:53.207119  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:53.216485  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:53.235950  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:53.249472  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:53.279855  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:53.374723  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:53.454142  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:53.549847  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 02:00:53.551745  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:53.554512  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:53.572525  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 02:00:53.586861  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:53.610448  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:53.612744  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:53.625588  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:53.628810  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:53.635182  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:53.707408  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:53.709985  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:53.743149  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 02:00:53.780874  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:53.781608  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:53.799687  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 02:00:53.810401  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:00:53.832298  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 02:00:53.841336  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:53.934514  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:53.961081  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 02:00:53.967520  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:00:53.971025  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 02:00:54.004297  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:54.005779  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:54.015254  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 02:00:54.029273  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:54.047639  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:54.050516  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:54.061069  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:54.196112  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 02:00:54.316112  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:54.340970  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:54.349911  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:54.381274  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 02:00:54.389240  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:54.397250  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:54.403513  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:54.410859  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 02:00:54.412822  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 02:00:54.426210  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:54.448084  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:54.470199  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:54.480464  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:54.507295  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:54.510831  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:54.520846  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:54.588665  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:54.605089  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 02:00:54.614034  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 02:00:54.614333  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:00:54.645910  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 02:00:54.876737  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:54.881803  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:54.888772  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:54.927105  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:54.933516  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:54.948019  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:54.957746  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:55.009712  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:55.012387  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:55.015635  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:55.058774  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 02:00:55.063462  6402 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.133:43379, user_credentials={real_user=slave}} blocked reactor thread for 54151.4us
I20240502 02:00:55.089139  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:55.097229  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:55.101987  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:55.138005  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:55.220861  6402 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 02:00:55.271739  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:00:55.313366  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 02:00:55.327821  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 02:00:55.350924  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:55.358397  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:55.383386  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:55.401551  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:55.403261  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:55.417688  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:55.429276  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:55.438028 24311 raft_consensus.cc:482] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 02:00:55.438230 24311 raft_consensus.cc:504] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 02:00:55.438992 24311 leader_election.cc:288] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 02:00:55.441277  6270 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:55.441573  6269 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:55.441658  6269 leader_election.cc:302] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 02:00:55.442198 24311 raft_consensus.cc:2726] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:00:55.500739  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:55.616780  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:55.644708  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:55.658210  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:55.686573  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:55.700842  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 02:00:55.727793  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:55.762812  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:55.847501  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:00:55.862784  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:55.926317  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 02:00:55.964974  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:56.027390  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:56.034225  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:56.041903  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 02:00:56.060087  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:56.061156  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:56.111172  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:56.144559  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:56.147240  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:00:56.149132  5999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 02:00:56.190465  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 02:00:56.342881  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 02:00:56.344219  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:56.376906  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 02:00:56.384117  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:56.386099  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:56.391026  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 02:00:56.403775  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 02:00:56.406283  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:56.408264  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:56.429392  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:56.430500  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:56.454344  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:56.454402  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:56.488881  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:56.514369  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:56.520102  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:56.521973  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:56.537595  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:56.577365  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 02:00:56.604775  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 02:00:56.725369  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:00:56.737435  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:00:56.792150  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:56.900732  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:00:56.901855  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 02:00:56.904706  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:56.905500  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 02:00:56.908291  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:56.909018  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:56.910302  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:56.936704  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:56.969928  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:56.975104  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:56.992681  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:57.009253  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:57.024026  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 02:00:57.053977  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:57.074103  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:57.087020  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 02:00:57.138736  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:57.158160  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:57.241788  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:57.299376  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 02:00:57.319522  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:57.324589  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:57.337132  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:57.339674  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:57.360846  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:57.397866  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:57.444465  6135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 02:00:57.545691  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:57.629446  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:00:57.674486  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:57.695433  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:57.705544  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:57.740381  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 02:00:57.742172  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:57.824007  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:57.846388  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 02:00:57.857316  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:57.877802  6269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240502 02:00:57.881364  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:57.883723  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:57.887687  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:57.889248  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:57.918253  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:57.924314  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:57.929831  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:58.011930 24912 raft_consensus.cc:482] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 02:00:58.012159 24912 raft_consensus.cc:504] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 02:00:58.012946 24912 leader_election.cc:288] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 02:00:58.023069  6269 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:58.023480  6270 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:58.023612  6270 leader_election.cc:302] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 02:00:58.024098 24912 raft_consensus.cc:2726] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 02:00:58.049304  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:58.091374  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 02:00:58.127915  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 02:00:58.248253  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:58.256536  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:00:58.270505  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:58.275614  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:58.288309  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:58.310057  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:58.339349  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:00:58.362541  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:58.364609  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:58.448702  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 02:00:58.526469  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:58.528700  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:58.547207  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:58.564143  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:58.567104  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:58.569928  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:58.574296  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:00:58.594959  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 02:00:58.624053  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 02:00:58.680323  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 02:00:58.720746  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 02:00:58.761696  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:58.835974  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:58.863353  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:58.866572  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:58.906692  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:58.907603  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:58.920143  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 02:00:58.928647  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 02:00:58.939271  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:00:58.940605  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:59.029733  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:00:59.060387  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:00:59.096648  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 02:00:59.145625  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:59.159305  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:59.175230  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:59.193626  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:59.202090  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:59.225589  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:59.241712  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:00:59.263346  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:00:59.282137  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 02:00:59.303629  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:59.316260  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:59.332273  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:00:59.332944  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:00:59.343997  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:00:59.350054  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:00:59.366497  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:59.416801  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:59.446694  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 02:00:59.619375  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:00:59.636919  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 02:00:59.648046  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:59.678120  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:59.690326  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:59.694080  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:59.738955  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:00:59.788661  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:00:59.790467  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 02:00:59.816159  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:00:59.889144  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 02:00:59.908622  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:00:59.925521  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:00:59.927520  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:00:59.948719  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:00:59.979882  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:00.005399  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:00.133785  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:01:00.223623  6401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 78 similar messages]
W20240502 02:01:00.248920  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 02:01:00.272708  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 02:01:00.274777  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:01:00.276899  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 02:01:00.306138  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:00.330236  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:00.337806  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:00.353349  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 02:01:00.373699  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:00.390470  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:00.412776  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:01:00.419466  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:01:00.580137  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:00.602706  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:00.604943  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:00.654354  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:00.677075  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:00.708010  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:00.708930  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:00.709700  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:00.715922  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:00.723162  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 02:01:00.756140  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:01:00.780344  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 02:01:00.815784  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:00.855648  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:00.869498  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 02:01:00.941165  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:01:00.972616  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:01:01.084909  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:01.086437  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:01.096490  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:01.101809  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:01.117293  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:01.136250  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:01.150964  5999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240502 02:01:01.179154  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:01:01.188604  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 02:01:01.192605  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 02:01:01.305603  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 02:01:01.347164  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 02:01:01.349900  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 02:01:01.383904  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 02:01:01.416558  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:01:01.431739  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:01.443322  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:01:01.471909  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:01:01.485481  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:01.486961  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:01.494302  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:01.531030  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:01.535085  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:01.558970  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:01.569933  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:01.584650  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:01.585728  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:01:01.605062  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 02:01:01.674472  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:01.703754  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:01:01.709278  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 02:01:01.726480  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:01.782302  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:01:01.836994  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:01:01.946231  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:01.948294  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:01.955646  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:01.999931  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:02.000891  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:02.015609  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:01:02.046500  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 02:01:02.057806  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 02:01:02.058930  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:01:02.149117  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 02:01:02.170390  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 02:01:02.223250  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:02.223788  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:02.230680  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:02.266511  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:02.302520  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:02.326596  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:02.460299  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:01:02.462044  6136 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240502 02:01:02.480687  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:02.489266  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:01:02.493870  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:01:02.509085  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:02.515923  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 02:01:02.524899  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:02.577754  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:02.578577  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:01:02.717489  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 02:01:02.767356  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 02:01:02.790163  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 02:01:02.811105  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:02.826763  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:02.829452  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:02.833575  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:02.864540  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:02.893276  6270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 54 similar messages]
W20240502 02:01:02.894376  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 02:01:02.927971  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:02.933698  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:01:02.977550  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:01:03.010246  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 02:01:03.047537  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:03.058523  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:03.065330  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:03.075662  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:03.094588  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:03.117067  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:03.311913  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:03.322450  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:03.337388  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:03.370273  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:03.370270  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:03.375078  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:01:03.390558  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:01:03.396284  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:01:03.512418  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 02:01:03.570230  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 02:01:03.582759  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:01:03.586416  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 02:01:03.660477  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:03.681931  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:03.686654  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:03.689615  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:03.703920  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:01:03.720764  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 02:01:03.730962  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:03.731979  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:03.752868  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:03.754101  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:03.805742  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 02:01:03.809471  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:03.833189  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:01:03.917376  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:01:03.933624  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:03.968761  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:03.975538  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 02:01:03.989451  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 02:01:04.024909  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 02:01:04.097394  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:01:04.098511  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 02:01:04.110837  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:04.122591  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:04.129626  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:04.142772  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:04.145395  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:04.169031  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:01:04.170765  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:01:04.172564  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:01:04.321907  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 02:01:04.339234  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 02:01:04.361958  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:04.384790  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:04.386915  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:04.395186  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:04.398742  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:04.421970  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:04.425098  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:01:04.446508  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 02:01:04.476547  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 02:01:04.607157  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:04.616171  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:04.626437  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:04.626978  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 02:01:04.628541  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:01:04.662935  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:04.705417  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:04.708559  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:04.918874  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:04.933840  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:04.939841  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:04.971975  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:01:04.973167  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:04.975116  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:04.992094  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:05.039160  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:01:05.057554  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 02:01:05.142181  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 02:01:05.155926  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:05.161665  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 02:01:05.172541  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:01:05.184386  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 02:01:05.205919  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:05.226799  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:05.236101  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:01:05.285048  6401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 80 similar messages]
I20240502 02:01:05.296422  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:05.358995  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:01:05.385195  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 02:01:05.407809  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 02:01:05.442898  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:01:05.464212  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:05.472586  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:05.498188  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:05.519675  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:05.522136  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:05.554770  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:05.707983  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 02:01:05.757616  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:05.760334  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 02:01:05.775451  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:05.784677  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:05.820464  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:05.824730  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:05.836910  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:01:05.837113  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:01:05.850267  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:05.965207  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 02:01:05.984007  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:06.041265  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:06.100387  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 02:01:06.106460  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:06.107391  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:06.109912  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:06.123035  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:06.146613  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 02:01:06.189401  6000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240502 02:01:06.190397  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 02:01:06.234041  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:01:06.248396  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 02:01:06.296209  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:01:06.303773  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 02:01:06.349196  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:06.372642  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:01:06.383487  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 02:01:06.388360  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:06.405284  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:01:06.415252  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 02:01:06.420159  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:06.430847  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 02:01:06.462061  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:06.554237  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 02:01:06.640805  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:01:06.697574  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:06.707832  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:06.724097  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 02:01:06.745867  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:06.763224  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 02:01:06.773522  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:01:06.789474  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:01:06.802213  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:06.810717  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:06.819268  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:06.840399  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:06.889348  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 02:01:06.889482  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 02:01:06.900732  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:06.904191  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 02:01:06.932632  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:06.941404  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:07.019389  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:01:07.079267  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:01:07.155826  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 02:01:07.207670  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:07.225800  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:07.240938  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:07.265901  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:07.266371  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:01:07.295429  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:07.324328  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:07.508374  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:07.509435  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:07.510344  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:07.514616  6135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 02:01:07.544911  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 02:01:07.545516  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:07.567070  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:07.609740  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 02:01:07.615938  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 02:01:07.617026  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:07.642453  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:01:07.692806  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:01:07.743775  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 02:01:07.747174  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:07.750842  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:07.763306  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:07.764113  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:07.819221  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 02:01:07.831203  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:07.845217  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 02:01:07.879403  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:07.968026  6269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240502 02:01:08.015440  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:01:08.030952  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:08.049014  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:08.050153  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:08.051442  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:08.069528  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:08.097455  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:08.123471  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 02:01:08.187333  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:01:08.221674  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 02:01:08.246138  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:08.248661  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:08.270613  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:08.278441  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:08.287696  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:08.313623  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:08.390349  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 02:01:08.439772  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 02:01:08.513139  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:08.534554  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:08.540661  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:08.548118  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:08.570652  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:08.584820  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:01:08.585680  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 02:01:08.616729  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:01:08.618297  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 02:01:08.624280  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 02:01:08.641685  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:08.685238  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 02:01:08.718623  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 02:01:08.768383 26065 raft_consensus.cc:482] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 02:01:08.768811 26065 raft_consensus.cc:504] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 02:01:08.778602 26065 leader_election.cc:288] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 02:01:08.780772  6000 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:01:08.781158  5999 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:01:08.781258  5999 leader_election.cc:302] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 02:01:08.783283 26065 raft_consensus.cc:2726] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:01:08.784385  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 02:01:08.867378  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:08.901497  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:08.902303  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:01:08.907141  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 02:01:08.912560  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:08.948612  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 02:01:08.978053  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:08.980903  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:09.098485  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:01:09.099313  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:01:09.150502  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 02:01:09.170661  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 02:01:09.203364  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:09.224799  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:09.225772  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:09.227260  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:09.273963  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:09.317726  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:09.328735  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 02:01:09.434011  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:09.442147  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 02:01:09.444221  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:09.470862  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:09.508208  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:09.544301  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:09.604167  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:09.604305  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 02:01:09.616201  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:01:09.673607  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:09.714007  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 02:01:09.719203  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:01:09.733798  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:09.740789  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:09.745543  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:09.746251  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:09.765321  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:10.045084  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:10.057982  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:10.075320  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:10.079448  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 02:01:10.114904  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:10.116461  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:01:10.134130  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:01:10.163745  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:10.177929  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 02:01:10.188040  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 02:01:10.207753  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 02:01:10.222719  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:01:10.246187  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:10.254992  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:10.271239  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:10.277725  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:10.278074  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:01:10.286191  6402 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 81 similar messages]
W20240502 02:01:10.290045  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 02:01:10.323532  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:10.352452 26687 raft_consensus.cc:482] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 02:01:10.352681 26687 raft_consensus.cc:504] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 02:01:10.353509 26687 leader_election.cc:288] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 02:01:10.354449  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 02:01:10.356398  6269 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:01:10.356698  6270 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:01:10.356803  6270 leader_election.cc:302] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 02:01:10.357430 26687 raft_consensus.cc:2726] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 02:01:10.425772  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:10.442972  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:10.451364  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:10.490191  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:10.500375  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:10.519482  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:10.532536  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:01:10.628063  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:01:10.706683  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 02:01:10.746248  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:10.757861  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:01:10.764797  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 02:01:10.766557  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:10.803009  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:10.832849  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:10.891072  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:10.909804  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 02:01:10.910995  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:01:10.990224  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:11.000597  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:11.013883  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:11.018630  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 02:01:11.037257  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:11.062960  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:11.065945  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:11.086452  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 02:01:11.087450  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 02:01:11.112538  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 02:01:11.163766  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:01:11.298545  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 02:01:11.307757  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:11.318600  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:11.318853  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:01:11.319382  5999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 72 similar messages]
I20240502 02:01:11.328640  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:11.332496  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:11.354483  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 02:01:11.363487  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 02:01:11.363907  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 02:01:11.378018  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:11.514396  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:01:11.553015  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:11.553871  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:11.555233  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:11.567308  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:11.567399  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:11.575191  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:11.584883  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:01:11.635166  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 02:01:11.696520  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 02:01:11.781644  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:11.802930  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:11.808873  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:11.818619  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:01:11.855360  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 02:01:11.855654  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 02:01:11.866302  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:11.894142  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 02:01:11.895212  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:11.926331  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 02:01:11.995623  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:01:12.001186  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:01:12.090117  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:12.091939  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:12.120807  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:12.121331  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:12.135874  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:12.143324  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:12.171026  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 02:01:12.284950  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:12.328955  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 02:01:12.333539  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:12.334631  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:12.340265  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:12.348718  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:12.364403  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:12.500016  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:01:12.518596  6135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 02:01:12.531805  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 02:01:12.553377  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:12.560797  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:12.564466  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:12.564802  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:01:12.585178  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 02:01:12.618252  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:12.621363  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:12.637010  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:12.790315  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 02:01:12.800310  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 02:01:12.915616  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:12.915789  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:12.950107  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:12.974166  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:12.979665  6269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 02:01:12.994378  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 02:01:13.021502  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:13.022701  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:01:13.025532  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:13.043025  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:13.061879  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:13.081766  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:13.105401  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:13.160084  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:01:13.166069  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:13.176590  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:13.300748  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 02:01:13.418300  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 02:01:13.425336  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:13.429988  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:13.432687  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:13.469486  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:13.478552  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:01:13.482605  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:01:13.536443  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 02:01:13.578998  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:13.586159  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 02:01:13.612231  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:01:13.646945  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 02:01:13.667537  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 02:01:13.669756  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:13.705216  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 02:01:13.726276  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:01:13.763978  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 02:01:13.771188  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:13.775626  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:13.807914  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 02:01:13.816681  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:13.819882  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:13.885043  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 02:01:13.967913  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 02:01:14.027107  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 02:01:14.029177  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:14.041997  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:14.042433  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:14.045881  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:14.053572  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:14.056016  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:01:14.059633  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:01:14.106711  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:01:14.135545  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 02:01:14.287250  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:14.327359  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 02:01:14.330044  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:14.361038  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 02:01:14.373327  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:14.376844  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 02:01:14.426788  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:14.444752  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:14.459470  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:14.467906  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:14.471073  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:14.484155  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:14.515259  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 02:01:14.546738  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:14.548257  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:01:14.560637  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:01:14.563369  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:14.687832  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 02:01:14.688244  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:01:14.726121  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 02:01:14.906340  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:14.909539  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:14.957777  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:01:14.969491  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 02:01:14.973876  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:01:14.982169  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 02:01:14.985702  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:15.006268  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:15.083101  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:15.111845  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:15.115463  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:15.133308  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:01:15.143764  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:01:15.166210  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:15.175364  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:15.206847  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:15.303815  6401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 79 similar messages]
W20240502 02:01:15.304934  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 02:01:15.337968  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 02:01:15.355021  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:15.359665  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:15.360432  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:15.372656  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:15.383173  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:15.399638  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:15.416857  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:01:15.607079  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 02:01:15.701265  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:15.711848  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:15.731781  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:15.733516  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:15.735157  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:15.748139  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:15.779523  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:15.780102  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:01:15.801491  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:15.803681  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:15.826054  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 02:01:15.838436  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:01:15.879313  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 02:01:15.883606  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:15.901754  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:15.913193  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 02:01:15.952307  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:01:16.017935  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 02:01:16.130405  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 02:01:16.151412  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:16.167587  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:16.168071  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 02:01:16.175091  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:16.175390 26817 raft_consensus.cc:482] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 02:01:16.175606 26817 raft_consensus.cc:504] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
W20240502 02:01:16.175747  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 02:01:16.176681 26817 leader_election.cc:288] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
I20240502 02:01:16.184192  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:16.191406  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:01:16.205107  5999 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:01:16.214031  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 02:01:16.227964  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:16.249924  6000 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:01:16.256647  6000 leader_election.cc:302] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 02:01:16.258234 26817 raft_consensus.cc:2726] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 02:01:16.297417  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:16.366789  6000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240502 02:01:16.367949  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 02:01:16.401628  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:16.414636  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 02:01:16.436457  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:16.463972  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:16.470669  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:16.496555  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:16.530117  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 02:01:16.530728  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 02:01:16.560779  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:16.577495  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:01:16.641328  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 02:01:16.696171  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:01:16.698323  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:16.703716  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:16.725210  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:16.732867  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:16.736208  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:16.762154  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 02:01:16.778122  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:01:16.847421  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 02:01:16.933054  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:16.952150  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:16.978499  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:01:16.983299  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 02:01:17.000522  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 02:01:17.001607  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:17.011736  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:17.038935  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:17.073560  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:01:17.094822  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 02:01:17.169951  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:17.228945  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:17.237012  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 02:01:17.237156  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:01:17.257977  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:17.272572  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:17.273123  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:17.282756  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:17.398175  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 02:01:17.457499  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 02:01:17.486317  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:17.487515  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:17.488075  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:17.500360  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:17.527438  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:17.563652  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:17.573139  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:17.575989  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 02:01:17.578985  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:17.582204  6136 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240502 02:01:17.585268  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:01:17.596961  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:01:17.660079  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:01:17.672225  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:01:17.737443  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:17.739159  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:17.967396  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 02:01:17.968271  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:17.978665  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:17.996269  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:01:18.009249  6269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 02:01:18.014582  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:01:18.023409  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:18.045289  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:18.058741  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:18.097878  6268 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.135:40543, user_credentials={real_user=slave}} blocked reactor thread for 44808.2us
W20240502 02:01:18.112433  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 02:01:18.131898  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 02:01:18.226876  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:01:18.244553  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:18.254140  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:18.255533  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:18.303987  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:18.306679  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:18.331167  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:18.354952  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 02:01:18.487831  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:18.501894  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:18.501909  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:18.503255  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:01:18.510497  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 02:01:18.535845  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:18.561237  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:18.590932  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 02:01:18.608381  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 02:01:18.616461  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:01:18.638168 27821 raft_consensus.cc:482] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 02:01:18.638389 27821 raft_consensus.cc:504] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 02:01:18.639209 27821 leader_election.cc:288] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 02:01:18.648195  6269 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:01:18.650089  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:01:18.660161  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 02:01:18.669111  6270 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:01:18.669328  6270 leader_election.cc:302] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 02:01:18.669701 27821 raft_consensus.cc:2726] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:01:18.780112  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 02:01:18.846412  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:18.872776  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:18.891716  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:18.893039  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 02:01:18.909811  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:18.923741  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:18.928279  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:18.972640  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:18.982224  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:18.986332  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:19.015513  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:01:19.026541  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 02:01:19.049974  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 02:01:19.059497  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:19.068172  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:01:19.070783  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 02:01:19.082840  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:19.124359  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 02:01:19.142642  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 02:01:19.263382  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:01:19.285140  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 02:01:19.287194  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:19.302001  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:01:19.309038  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 02:01:19.325191  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:19.326094  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:19.328459  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 02:01:19.378674  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:19.413542  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:19.526051  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:19.557623  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:19.583020  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:19.608966  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:19.611207  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:19.619887  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 02:01:19.624884  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:19.628567  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:01:19.651916  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:01:19.659282  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 02:01:19.720682  6000 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 02:01:19.782862  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:19.796811  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:19.801352  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:19.881805  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 02:01:19.885728  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:19.897600  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:19.914433  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:19.919848  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 02:01:20.065300  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:01:20.163224  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:20.167639  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:01:20.201828  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:20.220880  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:20.227689  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:20.252583  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:20.255316  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:20.259938  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 02:01:20.262116  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 02:01:20.270877  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:20.301028  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:20.312713  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:20.315295  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:20.315968  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:20.363941  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:20.436813  6401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 80 similar messages]
W20240502 02:01:20.468600  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 02:01:20.570250  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:01:20.596561  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 02:01:20.655388  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:20.683676  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:20.698120  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:20.709928  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:20.722903  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:20.730216  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:20.788066  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:01:20.878005  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:20.919819  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:20.920406  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:20.931185  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:20.944010  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:20.979589  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:21.029911  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:01:21.051450  6402 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 02:01:21.069626  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 02:01:21.080931  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 02:01:21.086266  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 02:01:21.122959  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:21.123778  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:21.153242  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:21.188724  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:21.200171  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:21.203661  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 02:01:21.228318  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:01:21.367555  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:01:21.432729  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:21.440661  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 02:01:21.451237  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:21.458019  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:21.478471  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:21.480854  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:21.505239  6000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240502 02:01:21.510006  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240502 02:01:21.513420  6000 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 02:01:21.521008  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:21.596509  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:01:21.641793  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 02:01:21.706367  6000 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 02:01:21.734205  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:21.740309  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:01:21.751806  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 02:01:21.752477  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 02:01:21.766481  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:21.769270  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:21.783933  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:21.807765  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 02:01:21.831450  6270 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 02:01:21.836959  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:21.855881  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 02:01:21.857617  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:01:21.904968 28305 raft_consensus.cc:998] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3: Attempting to remove follower 9db17f84d37944098cbeb9e0d8456efb from the Raft config. Reason: Leader has been unable to successfully communicate with peer 9db17f84d37944098cbeb9e0d8456efb for more than 300 seconds (300.391s)
I20240502 02:01:21.905920 28305 consensus_queue.cc:237] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 02:01:21.936533  8270 raft_consensus.cc:1261] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 02:01:21.937469 28052 consensus_queue.cc:1035] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 02:01:21.940163  6067 raft_consensus.cc:1261] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 02:01:21.940876 28052 consensus_queue.cc:1035] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 02:01:21.947397  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 02:01:21.959277 28052 raft_consensus.cc:2932] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 02:01:21.966840  6066 raft_consensus.cc:2932] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 02:01:21.968025  5912 catalog_manager.cc:5484] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 reported cstate change: config changed from index 6 to 7, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) evicted. New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 02:01:21.972267  8269 raft_consensus.cc:2932] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 02:01:22.024858 27986 raft_consensus.cc:482] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 02:01:22.025076 27986 raft_consensus.cc:504] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 02:01:22.025822 27986 leader_election.cc:288] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 02:01:22.028915  5999 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:01:22.030484  6000 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:01:22.030622  6000 leader_election.cc:302] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 02:01:22.032446 27986 raft_consensus.cc:2726] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:01:22.033035  5907 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:01:22.036870  5907 catalog_manager.cc:4573] TS 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet afa5561f8c0e4c0fa9bbbd5da61348c7: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:01:22.053349  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:22.074497  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:22.084126 27986 raft_consensus.cc:998] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9: Attempting to remove follower 9db17f84d37944098cbeb9e0d8456efb from the Raft config. Reason: Leader has been unable to successfully communicate with peer 9db17f84d37944098cbeb9e0d8456efb for more than 300 seconds (300.403s)
I20240502 02:01:22.084293 28490 raft_consensus.cc:998] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9: Attempting to remove follower 9db17f84d37944098cbeb9e0d8456efb from the Raft config. Reason: Leader has been unable to successfully communicate with peer 9db17f84d37944098cbeb9e0d8456efb for more than 300 seconds (300.161s)
I20240502 02:01:22.085161 28490 consensus_queue.cc:237] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } }
I20240502 02:01:22.085199 27986 consensus_queue.cc:237] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 02:01:22.099041  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:22.116168 28493 raft_consensus.cc:998] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9: Attempting to remove follower 9db17f84d37944098cbeb9e0d8456efb from the Raft config. Reason: Leader has been unable to successfully communicate with peer 9db17f84d37944098cbeb9e0d8456efb for more than 300 seconds (300.104s)
I20240502 02:01:22.117130  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:22.117113 28493 consensus_queue.cc:237] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 02:01:22.127295  8269 raft_consensus.cc:1261] T 5be45a68cff24a75a03b2fcd4b7851f9 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 02:01:22.129741 28496 consensus_queue.cc:1035] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 02:01:22.133388  8270 raft_consensus.cc:1261] T 99e74a49b42745888058db67049dc682 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 02:01:22.134164 28493 consensus_queue.cc:1035] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 02:01:22.142001  6335 raft_consensus.cc:1261] T 899963a8260a42c4b73fb80eda2ecaa9 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:01:22.144229 27986 consensus_queue.cc:1035] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:01:22.156419  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:22.168334  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:22.195601 28502 raft_consensus.cc:998] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399: Attempting to remove follower 9db17f84d37944098cbeb9e0d8456efb from the Raft config. Reason: Leader has been unable to successfully communicate with peer 9db17f84d37944098cbeb9e0d8456efb for more than 300 seconds (300.417s)
I20240502 02:01:22.197187 28502 consensus_queue.cc:237] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 02:01:22.197535 28508 raft_consensus.cc:998] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399: Attempting to remove follower 9db17f84d37944098cbeb9e0d8456efb from the Raft config. Reason: Leader has been unable to successfully communicate with peer 9db17f84d37944098cbeb9e0d8456efb for more than 300 seconds (300.207s)
I20240502 02:01:22.199836 28508 consensus_queue.cc:237] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 02:01:22.206007 28496 raft_consensus.cc:2932] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } }
I20240502 02:01:22.210732  6334 raft_consensus.cc:2932] T 899963a8260a42c4b73fb80eda2ecaa9 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } }
I20240502 02:01:22.217167  5922 catalog_manager.cc:5484] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 reported cstate change: config changed from index 5 to 6, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) evicted. New cstate: current_term: 1 leader_uuid: "218bc8c1f9ce46059898fc738179adf9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } }
W20240502 02:01:22.230870  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 02:01:22.242158 28496 raft_consensus.cc:2932] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 02:01:22.244912  8270 raft_consensus.cc:2932] T 99e74a49b42745888058db67049dc682 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 02:01:22.252357 27986 raft_consensus.cc:2932] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
W20240502 02:01:22.257355  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:01:22.257217  6066 consensus_queue.cc:237] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 02:01:22.265475  5922 catalog_manager.cc:5484] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 reported cstate change: config changed from index 7 to 8, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) evicted. New cstate: current_term: 1 leader_uuid: "218bc8c1f9ce46059898fc738179adf9" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 02:01:22.266157  5922 catalog_manager.cc:5484] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 reported cstate change: config changed from index 7 to 8, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) evicted. New cstate: current_term: 1 leader_uuid: "218bc8c1f9ce46059898fc738179adf9" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
W20240502 02:01:22.275548  6001 consensus_peers.cc:485] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197): Couldn't send request to peer a2c90ee440b44b708bccc2e2d5385cc0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 899963a8260a42c4b73fb80eda2ecaa9. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:01:22.275516  8269 raft_consensus.cc:2932] T 5be45a68cff24a75a03b2fcd4b7851f9 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 02:01:22.276098  6067 consensus_queue.cc:237] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 02:01:22.276345  6066 consensus_queue.cc:237] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
W20240502 02:01:22.279500  6001 consensus_peers.cc:485] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 -> Peer a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197): Couldn't send request to peer a2c90ee440b44b708bccc2e2d5385cc0. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:01:22.279592  8268 raft_consensus.cc:1261] T 99e74a49b42745888058db67049dc682 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 02:01:22.281049 28496 consensus_queue.cc:1035] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 02:01:22.286634 28496 raft_consensus.cc:2932] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } }
I20240502 02:01:22.286726  8268 raft_consensus.cc:2932] T 99e74a49b42745888058db67049dc682 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } }
W20240502 02:01:22.290244  5998 consensus_peers.cc:485] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867): Couldn't send request to peer 07cdb213c77c4ffda1f999b66b725399. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:01:22.291179  6335 raft_consensus.cc:1261] T 899963a8260a42c4b73fb80eda2ecaa9 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 02:01:22.291388  5907 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 99e74a49b42745888058db67049dc682 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 02:01:22.292044 27986 consensus_queue.cc:1035] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 02:01:22.293272  5922 catalog_manager.cc:5484] T 99e74a49b42745888058db67049dc682 P 436822a397664c54b8d1b9eb3bc6cff4 reported cstate change: config changed from index 8 to 9, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) added. New cstate: current_term: 1 leader_uuid: "218bc8c1f9ce46059898fc738179adf9" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } }
I20240502 02:01:22.296815  8001 raft_consensus.cc:1261] T 6984d638991442e3adaed1b24bee95e9 P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 02:01:22.297456  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 02:01:22.297734 28372 consensus_queue.cc:1035] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 02:01:22.299959  6066 raft_consensus.cc:1261] T 6984d638991442e3adaed1b24bee95e9 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 02:01:22.301174 28508 consensus_queue.cc:1035] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 02:01:22.303812 28493 raft_consensus.cc:2932] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } }
I20240502 02:01:22.308998  5907 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 899963a8260a42c4b73fb80eda2ecaa9 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 02:01:22.310638  6334 raft_consensus.cc:2932] T 899963a8260a42c4b73fb80eda2ecaa9 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } }
I20240502 02:01:22.310892  5922 catalog_manager.cc:5484] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9 reported cstate change: config changed from index 6 to 7, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) added. New cstate: current_term: 1 leader_uuid: "218bc8c1f9ce46059898fc738179adf9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } }
I20240502 02:01:22.316840  8270 raft_consensus.cc:1261] T 5be45a68cff24a75a03b2fcd4b7851f9 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Refusing update from remote peer 218bc8c1f9ce46059898fc738179adf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 02:01:22.317868 28493 consensus_queue.cc:1035] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 02:01:22.320741 27986 raft_consensus.cc:2932] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } }
I20240502 02:01:22.322983  8268 raft_consensus.cc:2932] T 5be45a68cff24a75a03b2fcd4b7851f9 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } }
I20240502 02:01:22.329288  5921 catalog_manager.cc:5484] T 5be45a68cff24a75a03b2fcd4b7851f9 P 436822a397664c54b8d1b9eb3bc6cff4 reported cstate change: config changed from index 8 to 9, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) added. New cstate: current_term: 1 leader_uuid: "218bc8c1f9ce46059898fc738179adf9" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } }
W20240502 02:01:22.336294  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:01:22.337138  7999 raft_consensus.cc:1261] T 0cecd1b14e474dee825fae040ce1b31e P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 02:01:22.337363  6335 raft_consensus.cc:1261] T 0cecd1b14e474dee825fae040ce1b31e P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 02:01:22.337714 28502 consensus_queue.cc:1035] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 02:01:22.338407 28508 consensus_queue.cc:1035] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 02:01:22.342051 28508 raft_consensus.cc:2932] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 02:01:22.342345  7999 raft_consensus.cc:2932] T 0cecd1b14e474dee825fae040ce1b31e P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 02:01:22.344771 28052 raft_consensus.cc:998] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3: Attempting to remove follower 9db17f84d37944098cbeb9e0d8456efb from the Raft config. Reason: Leader has been unable to successfully communicate with peer 9db17f84d37944098cbeb9e0d8456efb for more than 300 seconds (300.393s)
I20240502 02:01:22.345419 28052 consensus_queue.cc:237] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 02:01:22.352536  5907 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 5be45a68cff24a75a03b2fcd4b7851f9 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 02:01:22.352284  5921 catalog_manager.cc:5484] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 reported cstate change: config changed from index 6 to 7, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) evicted. New cstate: current_term: 1 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 02:01:22.355955  6200 raft_consensus.cc:1261] T 7f2519b0c6404799ad84d120b45a7ea4 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:01:22.357692 28372 raft_consensus.cc:2932] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 02:01:22.361008 28052 consensus_queue.cc:1035] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:01:22.351007  6335 raft_consensus.cc:2932] T 0cecd1b14e474dee825fae040ce1b31e P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 02:01:22.363642  6066 raft_consensus.cc:2932] T 6984d638991442e3adaed1b24bee95e9 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 02:01:22.365063  5922 catalog_manager.cc:5484] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 reported cstate change: config changed from index 7 to 8, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) evicted. New cstate: current_term: 1 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 02:01:22.379916  7999 raft_consensus.cc:2932] T 6984d638991442e3adaed1b24bee95e9 P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 02:01:22.390302 28469 raft_consensus.cc:2932] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } }
I20240502 02:01:22.395522  6203 raft_consensus.cc:2932] T 7f2519b0c6404799ad84d120b45a7ea4 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } }
I20240502 02:01:22.402369  5921 catalog_manager.cc:5484] T 7f2519b0c6404799ad84d120b45a7ea4 P a2c90ee440b44b708bccc2e2d5385cc0 reported cstate change: config changed from index 5 to 6, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) evicted. New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } }
I20240502 02:01:22.417515  6335 consensus_queue.cc:237] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 02:01:22.419993  6203 raft_consensus.cc:1261] T 7f2519b0c6404799ad84d120b45a7ea4 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 02:01:22.420639 28469 consensus_queue.cc:1035] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 02:01:22.423120  6268 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867): Couldn't send request to peer 07cdb213c77c4ffda1f999b66b725399. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7f2519b0c6404799ad84d120b45a7ea4. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:01:22.423730 28469 raft_consensus.cc:2932] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } }
I20240502 02:01:22.428910  5907 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 7f2519b0c6404799ad84d120b45a7ea4 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 02:01:22.430315  6203 raft_consensus.cc:2932] T 7f2519b0c6404799ad84d120b45a7ea4 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } }
I20240502 02:01:22.437258  5922 catalog_manager.cc:5484] T 7f2519b0c6404799ad84d120b45a7ea4 P a2c90ee440b44b708bccc2e2d5385cc0 reported cstate change: config changed from index 6 to 7, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) added. New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } }
W20240502 02:01:22.573104  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 02:01:22.574050  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:22.576360  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:22.595288  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:22.595413  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:22.603343  6136 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 56 similar messages]
W20240502 02:01:22.628602  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 02:01:22.648036  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:01:22.656562  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:22.656839  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:22.665115  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:22.701720  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:22.704324 28372 raft_consensus.cc:998] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399: Attempting to remove follower 9db17f84d37944098cbeb9e0d8456efb from the Raft config. Reason: Leader has been unable to successfully communicate with peer 9db17f84d37944098cbeb9e0d8456efb for more than 300 seconds (300.503s)
I20240502 02:01:22.705032 28372 consensus_queue.cc:237] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 02:01:22.730377  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:22.736784  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:22.742604  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:22.744760  6200 raft_consensus.cc:1261] T f309b40931dd444091869119eab33bbc P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:01:22.755712 28372 consensus_queue.cc:1035] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:01:22.762876  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:22.812233 28601 ts_tablet_manager.cc:902] T 99e74a49b42745888058db67049dc682 P 07cdb213c77c4ffda1f999b66b725399: Initiating tablet copy from peer 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227)
I20240502 02:01:22.844523 28508 raft_consensus.cc:2932] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } }
I20240502 02:01:22.850405  6200 raft_consensus.cc:2932] T f309b40931dd444091869119eab33bbc P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } }
I20240502 02:01:22.853755  5922 catalog_manager.cc:5484] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 reported cstate change: config changed from index 5 to 6, VOTER 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134) evicted. New cstate: current_term: 1 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } }
I20240502 02:01:22.874087  6469 consensus_queue.cc:237] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 02:01:22.878363  6200 raft_consensus.cc:1261] T f309b40931dd444091869119eab33bbc P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 02:01:22.879369  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133:43379): Couldn't send request to peer 60298ffea1dd4c25a38765c2ec9ec9d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f309b40931dd444091869119eab33bbc. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:01:22.879945 28508 consensus_queue.cc:1035] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 02:01:22.895185  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:01:22.904742 28372 raft_consensus.cc:2932] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 02:01:22.908471  6200 raft_consensus.cc:2932] T f309b40931dd444091869119eab33bbc P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 02:01:22.909902 28601 tablet_copy_client.cc:308] T 99e74a49b42745888058db67049dc682 P 07cdb213c77c4ffda1f999b66b725399: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.129:36227
I20240502 02:01:22.916462  5922 catalog_manager.cc:5484] T f309b40931dd444091869119eab33bbc P a2c90ee440b44b708bccc2e2d5385cc0 reported cstate change: config changed from index 6 to 7, VOTER 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133) added. New cstate: current_term: 1 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 02:01:22.920253  5905 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet f309b40931dd444091869119eab33bbc with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 02:01:22.935374 28612 ts_tablet_manager.cc:902] T 7f2519b0c6404799ad84d120b45a7ea4 P 07cdb213c77c4ffda1f999b66b725399: Initiating tablet copy from peer 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245)
I20240502 02:01:22.937075 28612 tablet_copy_client.cc:308] T 7f2519b0c6404799ad84d120b45a7ea4 P 07cdb213c77c4ffda1f999b66b725399: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.131:44245
I20240502 02:01:22.938161  6357 tablet_copy_service.cc:140] P 94ffd7aaf85b4e17881241b8396fa2f3: Received BeginTabletCopySession request for tablet 7f2519b0c6404799ad84d120b45a7ea4 from peer 07cdb213c77c4ffda1f999b66b725399 ({username='slave'} at 127.25.109.132:33439)
I20240502 02:01:22.938323  6357 tablet_copy_service.cc:161] P 94ffd7aaf85b4e17881241b8396fa2f3: Beginning new tablet copy session on tablet 7f2519b0c6404799ad84d120b45a7ea4 from peer 07cdb213c77c4ffda1f999b66b725399 at {username='slave'} at 127.25.109.132:33439: session id = 07cdb213c77c4ffda1f999b66b725399-7f2519b0c6404799ad84d120b45a7ea4
I20240502 02:01:22.942862  6357 tablet_copy_source_session.cc:205] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 02:01:22.944814 28607 ts_tablet_manager.cc:902] T 899963a8260a42c4b73fb80eda2ecaa9 P a2c90ee440b44b708bccc2e2d5385cc0: Initiating tablet copy from peer 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227)
I20240502 02:01:22.945423 28612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f2519b0c6404799ad84d120b45a7ea4. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 02:01:22.946110  5998 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.135:40543, user_credentials={real_user=slave}} blocked reactor thread for 68499.7us
W20240502 02:01:22.950670  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 02:01:22.959826 28612 tablet_copy_client.cc:783] T 7f2519b0c6404799ad84d120b45a7ea4 P 07cdb213c77c4ffda1f999b66b725399: tablet copy: Starting download of 5 data blocks...
I20240502 02:01:22.962985 28618 ts_tablet_manager.cc:902] T 5be45a68cff24a75a03b2fcd4b7851f9 P a2c90ee440b44b708bccc2e2d5385cc0: Initiating tablet copy from peer 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227)
W20240502 02:01:22.964819  6000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.131:44245, user_credentials={real_user=slave}} blocked reactor thread for 88877.5us
I20240502 02:01:22.965409  6089 tablet_copy_service.cc:140] P 218bc8c1f9ce46059898fc738179adf9: Received BeginTabletCopySession request for tablet 99e74a49b42745888058db67049dc682 from peer 07cdb213c77c4ffda1f999b66b725399 ({username='slave'} at 127.25.109.132:50167)
I20240502 02:01:22.965601  6089 tablet_copy_service.cc:161] P 218bc8c1f9ce46059898fc738179adf9: Beginning new tablet copy session on tablet 99e74a49b42745888058db67049dc682 from peer 07cdb213c77c4ffda1f999b66b725399 at {username='slave'} at 127.25.109.132:50167: session id = 07cdb213c77c4ffda1f999b66b725399-99e74a49b42745888058db67049dc682
I20240502 02:01:22.970369  6089 tablet_copy_source_session.cc:205] T 99e74a49b42745888058db67049dc682 P 218bc8c1f9ce46059898fc738179adf9: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 02:01:22.971853 28601 ts_tablet_manager.cc:1875] T 99e74a49b42745888058db67049dc682 P 07cdb213c77c4ffda1f999b66b725399: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 02:01:22.976990 28607 tablet_copy_client.cc:308] T 899963a8260a42c4b73fb80eda2ecaa9 P a2c90ee440b44b708bccc2e2d5385cc0: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.129:36227
I20240502 02:01:22.980670 28612 tablet_copy_client.cc:647] T 7f2519b0c6404799ad84d120b45a7ea4 P 07cdb213c77c4ffda1f999b66b725399: tablet copy: Starting download of 1 WAL segments...
I20240502 02:01:22.983806 28601 ts_tablet_manager.cc:1888] T 99e74a49b42745888058db67049dc682 P 07cdb213c77c4ffda1f999b66b725399: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240502 02:01:22.983942 28612 tablet_copy_client.cc:523] T 7f2519b0c6404799ad84d120b45a7ea4 P 07cdb213c77c4ffda1f999b66b725399: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:01:22.984396 28601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99e74a49b42745888058db67049dc682. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:22.988925 28612 tablet_bootstrap.cc:492] T 7f2519b0c6404799ad84d120b45a7ea4 P 07cdb213c77c4ffda1f999b66b725399: Bootstrap starting.
I20240502 02:01:22.988965  6089 tablet_copy_service.cc:140] P 218bc8c1f9ce46059898fc738179adf9: Received BeginTabletCopySession request for tablet 899963a8260a42c4b73fb80eda2ecaa9 from peer a2c90ee440b44b708bccc2e2d5385cc0 ({username='slave'} at 127.25.109.130:60995)
I20240502 02:01:22.989123  6089 tablet_copy_service.cc:161] P 218bc8c1f9ce46059898fc738179adf9: Beginning new tablet copy session on tablet 899963a8260a42c4b73fb80eda2ecaa9 from peer a2c90ee440b44b708bccc2e2d5385cc0 at {username='slave'} at 127.25.109.130:60995: session id = a2c90ee440b44b708bccc2e2d5385cc0-899963a8260a42c4b73fb80eda2ecaa9
I20240502 02:01:22.993362  6089 tablet_copy_source_session.cc:205] T 899963a8260a42c4b73fb80eda2ecaa9 P 218bc8c1f9ce46059898fc738179adf9: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 02:01:22.995321 28601 tablet_copy_client.cc:783] T 99e74a49b42745888058db67049dc682 P 07cdb213c77c4ffda1f999b66b725399: tablet copy: Starting download of 5 data blocks...
I20240502 02:01:22.995638 28607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 899963a8260a42c4b73fb80eda2ecaa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:23.002183 28601 tablet_copy_client.cc:647] T 99e74a49b42745888058db67049dc682 P 07cdb213c77c4ffda1f999b66b725399: tablet copy: Starting download of 1 WAL segments...
I20240502 02:01:23.004344 28601 tablet_copy_client.cc:523] T 99e74a49b42745888058db67049dc682 P 07cdb213c77c4ffda1f999b66b725399: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:01:23.009120 28607 tablet_copy_client.cc:783] T 899963a8260a42c4b73fb80eda2ecaa9 P a2c90ee440b44b708bccc2e2d5385cc0: tablet copy: Starting download of 5 data blocks...
I20240502 02:01:23.011039 28601 tablet_bootstrap.cc:492] T 99e74a49b42745888058db67049dc682 P 07cdb213c77c4ffda1f999b66b725399: Bootstrap starting.
I20240502 02:01:23.023921 28607 tablet_copy_client.cc:647] T 899963a8260a42c4b73fb80eda2ecaa9 P a2c90ee440b44b708bccc2e2d5385cc0: tablet copy: Starting download of 1 WAL segments...
I20240502 02:01:23.025780 28607 tablet_copy_client.cc:523] T 899963a8260a42c4b73fb80eda2ecaa9 P a2c90ee440b44b708bccc2e2d5385cc0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:01:23.037938 28607 tablet_bootstrap.cc:492] T 899963a8260a42c4b73fb80eda2ecaa9 P a2c90ee440b44b708bccc2e2d5385cc0: Bootstrap starting.
I20240502 02:01:23.041630 28618 tablet_copy_client.cc:308] T 5be45a68cff24a75a03b2fcd4b7851f9 P a2c90ee440b44b708bccc2e2d5385cc0: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.129:36227
I20240502 02:01:23.042460  6089 tablet_copy_service.cc:140] P 218bc8c1f9ce46059898fc738179adf9: Received BeginTabletCopySession request for tablet 5be45a68cff24a75a03b2fcd4b7851f9 from peer a2c90ee440b44b708bccc2e2d5385cc0 ({username='slave'} at 127.25.109.130:60995)
I20240502 02:01:23.042649  6089 tablet_copy_service.cc:161] P 218bc8c1f9ce46059898fc738179adf9: Beginning new tablet copy session on tablet 5be45a68cff24a75a03b2fcd4b7851f9 from peer a2c90ee440b44b708bccc2e2d5385cc0 at {username='slave'} at 127.25.109.130:60995: session id = a2c90ee440b44b708bccc2e2d5385cc0-5be45a68cff24a75a03b2fcd4b7851f9
I20240502 02:01:23.046216  6089 tablet_copy_source_session.cc:205] T 5be45a68cff24a75a03b2fcd4b7851f9 P 218bc8c1f9ce46059898fc738179adf9: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 02:01:23.047567 28618 ts_tablet_manager.cc:1875] T 5be45a68cff24a75a03b2fcd4b7851f9 P a2c90ee440b44b708bccc2e2d5385cc0: Deleting tablet data with delete state TABLET_DATA_COPYING
W20240502 02:01:23.051999  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 02:01:23.066113 28601 tablet_bootstrap.cc:492] T 99e74a49b42745888058db67049dc682 P 07cdb213c77c4ffda1f999b66b725399: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:01:23.066620 28601 tablet_bootstrap.cc:492] T 99e74a49b42745888058db67049dc682 P 07cdb213c77c4ffda1f999b66b725399: Bootstrap complete.
I20240502 02:01:23.067025 28601 ts_tablet_manager.cc:1367] T 99e74a49b42745888058db67049dc682 P 07cdb213c77c4ffda1f999b66b725399: Time spent bootstrapping tablet: real 0.056s	user 0.023s	sys 0.012s
I20240502 02:01:23.067895 28601 raft_consensus.cc:348] T 99e74a49b42745888058db67049dc682 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 02:01:23.068150 28601 raft_consensus.cc:729] T 99e74a49b42745888058db67049dc682 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07cdb213c77c4ffda1f999b66b725399, State: Initialized, Role: FOLLOWER
I20240502 02:01:23.068499 28601 consensus_queue.cc:260] T 99e74a49b42745888058db67049dc682 P 07cdb213c77c4ffda1f999b66b725399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 02:01:23.070020 28601 ts_tablet_manager.cc:1398] T 99e74a49b42745888058db67049dc682 P 07cdb213c77c4ffda1f999b66b725399: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 02:01:23.071164  6089 tablet_copy_service.cc:342] P 218bc8c1f9ce46059898fc738179adf9: Request end of tablet copy session 07cdb213c77c4ffda1f999b66b725399-99e74a49b42745888058db67049dc682 received from {username='slave'} at 127.25.109.132:50167
I20240502 02:01:23.071295  6089 tablet_copy_service.cc:433] P 218bc8c1f9ce46059898fc738179adf9: ending tablet copy session 07cdb213c77c4ffda1f999b66b725399-99e74a49b42745888058db67049dc682 on tablet 99e74a49b42745888058db67049dc682 with peer 07cdb213c77c4ffda1f999b66b725399
I20240502 02:01:23.071343 28618 ts_tablet_manager.cc:1888] T 5be45a68cff24a75a03b2fcd4b7851f9 P a2c90ee440b44b708bccc2e2d5385cc0: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 02:01:23.071579 28618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5be45a68cff24a75a03b2fcd4b7851f9. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 02:01:23.072094  5907 catalog_manager.cc:4573] TS 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 7f2519b0c6404799ad84d120b45a7ea4: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:01:23.085794 28618 tablet_copy_client.cc:783] T 5be45a68cff24a75a03b2fcd4b7851f9 P a2c90ee440b44b708bccc2e2d5385cc0: tablet copy: Starting download of 5 data blocks...
I20240502 02:01:23.093716 28618 tablet_copy_client.cc:647] T 5be45a68cff24a75a03b2fcd4b7851f9 P a2c90ee440b44b708bccc2e2d5385cc0: tablet copy: Starting download of 1 WAL segments...
I20240502 02:01:23.095649 28618 tablet_copy_client.cc:523] T 5be45a68cff24a75a03b2fcd4b7851f9 P a2c90ee440b44b708bccc2e2d5385cc0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:01:23.097024 28607 tablet_bootstrap.cc:492] T 899963a8260a42c4b73fb80eda2ecaa9 P a2c90ee440b44b708bccc2e2d5385cc0: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:01:23.097512 28607 tablet_bootstrap.cc:492] T 899963a8260a42c4b73fb80eda2ecaa9 P a2c90ee440b44b708bccc2e2d5385cc0: Bootstrap complete.
I20240502 02:01:23.097839 28607 ts_tablet_manager.cc:1367] T 899963a8260a42c4b73fb80eda2ecaa9 P a2c90ee440b44b708bccc2e2d5385cc0: Time spent bootstrapping tablet: real 0.060s	user 0.022s	sys 0.007s
I20240502 02:01:23.100029 28618 tablet_bootstrap.cc:492] T 5be45a68cff24a75a03b2fcd4b7851f9 P a2c90ee440b44b708bccc2e2d5385cc0: Bootstrap starting.
I20240502 02:01:23.112110 28612 tablet_bootstrap.cc:492] T 7f2519b0c6404799ad84d120b45a7ea4 P 07cdb213c77c4ffda1f999b66b725399: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:01:23.112519 28612 tablet_bootstrap.cc:492] T 7f2519b0c6404799ad84d120b45a7ea4 P 07cdb213c77c4ffda1f999b66b725399: Bootstrap complete.
I20240502 02:01:23.112785 28612 ts_tablet_manager.cc:1367] T 7f2519b0c6404799ad84d120b45a7ea4 P 07cdb213c77c4ffda1f999b66b725399: Time spent bootstrapping tablet: real 0.124s	user 0.057s	sys 0.004s
I20240502 02:01:23.098695 28607 raft_consensus.cc:348] T 899963a8260a42c4b73fb80eda2ecaa9 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 02:01:23.113363 28607 raft_consensus.cc:729] T 899963a8260a42c4b73fb80eda2ecaa9 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c90ee440b44b708bccc2e2d5385cc0, State: Initialized, Role: FOLLOWER
I20240502 02:01:23.113576 28612 raft_consensus.cc:348] T 7f2519b0c6404799ad84d120b45a7ea4 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 02:01:23.113786 28612 raft_consensus.cc:729] T 7f2519b0c6404799ad84d120b45a7ea4 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07cdb213c77c4ffda1f999b66b725399, State: Initialized, Role: FOLLOWER
I20240502 02:01:23.114003 28612 consensus_queue.cc:260] T 7f2519b0c6404799ad84d120b45a7ea4 P 07cdb213c77c4ffda1f999b66b725399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 02:01:23.113992 28607 consensus_queue.cc:260] T 899963a8260a42c4b73fb80eda2ecaa9 P a2c90ee440b44b708bccc2e2d5385cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 02:01:23.114514 28612 ts_tablet_manager.cc:1398] T 7f2519b0c6404799ad84d120b45a7ea4 P 07cdb213c77c4ffda1f999b66b725399: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 02:01:23.115374  6357 tablet_copy_service.cc:342] P 94ffd7aaf85b4e17881241b8396fa2f3: Request end of tablet copy session 07cdb213c77c4ffda1f999b66b725399-7f2519b0c6404799ad84d120b45a7ea4 received from {username='slave'} at 127.25.109.132:33439
I20240502 02:01:23.115496  6357 tablet_copy_service.cc:433] P 94ffd7aaf85b4e17881241b8396fa2f3: ending tablet copy session 07cdb213c77c4ffda1f999b66b725399-7f2519b0c6404799ad84d120b45a7ea4 on tablet 7f2519b0c6404799ad84d120b45a7ea4 with peer 07cdb213c77c4ffda1f999b66b725399
I20240502 02:01:23.116650 28607 ts_tablet_manager.cc:1398] T 899963a8260a42c4b73fb80eda2ecaa9 P a2c90ee440b44b708bccc2e2d5385cc0: Time spent starting tablet: real 0.019s	user 0.003s	sys 0.001s
I20240502 02:01:23.119071  6089 tablet_copy_service.cc:342] P 218bc8c1f9ce46059898fc738179adf9: Request end of tablet copy session a2c90ee440b44b708bccc2e2d5385cc0-899963a8260a42c4b73fb80eda2ecaa9 received from {username='slave'} at 127.25.109.130:60995
I20240502 02:01:23.119211  6089 tablet_copy_service.cc:433] P 218bc8c1f9ce46059898fc738179adf9: ending tablet copy session a2c90ee440b44b708bccc2e2d5385cc0-899963a8260a42c4b73fb80eda2ecaa9 on tablet 899963a8260a42c4b73fb80eda2ecaa9 with peer a2c90ee440b44b708bccc2e2d5385cc0
I20240502 02:01:23.139000 28618 tablet_bootstrap.cc:492] T 5be45a68cff24a75a03b2fcd4b7851f9 P a2c90ee440b44b708bccc2e2d5385cc0: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:01:23.139428 28618 tablet_bootstrap.cc:492] T 5be45a68cff24a75a03b2fcd4b7851f9 P a2c90ee440b44b708bccc2e2d5385cc0: Bootstrap complete.
I20240502 02:01:23.139719 28618 ts_tablet_manager.cc:1367] T 5be45a68cff24a75a03b2fcd4b7851f9 P a2c90ee440b44b708bccc2e2d5385cc0: Time spent bootstrapping tablet: real 0.040s	user 0.014s	sys 0.009s
I20240502 02:01:23.140586 28618 raft_consensus.cc:348] T 5be45a68cff24a75a03b2fcd4b7851f9 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 02:01:23.140798 28618 raft_consensus.cc:729] T 5be45a68cff24a75a03b2fcd4b7851f9 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c90ee440b44b708bccc2e2d5385cc0, State: Initialized, Role: FOLLOWER
I20240502 02:01:23.141021 28618 consensus_queue.cc:260] T 5be45a68cff24a75a03b2fcd4b7851f9 P a2c90ee440b44b708bccc2e2d5385cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 02:01:23.141753 28618 ts_tablet_manager.cc:1398] T 5be45a68cff24a75a03b2fcd4b7851f9 P a2c90ee440b44b708bccc2e2d5385cc0: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.004s
I20240502 02:01:23.142544  6089 tablet_copy_service.cc:342] P 218bc8c1f9ce46059898fc738179adf9: Request end of tablet copy session a2c90ee440b44b708bccc2e2d5385cc0-5be45a68cff24a75a03b2fcd4b7851f9 received from {username='slave'} at 127.25.109.130:60995
I20240502 02:01:23.142699  6089 tablet_copy_service.cc:433] P 218bc8c1f9ce46059898fc738179adf9: ending tablet copy session a2c90ee440b44b708bccc2e2d5385cc0-5be45a68cff24a75a03b2fcd4b7851f9 on tablet 5be45a68cff24a75a03b2fcd4b7851f9 with peer a2c90ee440b44b708bccc2e2d5385cc0
W20240502 02:01:23.179214  6269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240502 02:01:23.225668  6515 maintenance_manager.cc:419] P 07cdb213c77c4ffda1f999b66b725399: Scheduling UndoDeltaBlockGCOp(7f2519b0c6404799ad84d120b45a7ea4): 4921 bytes on disk
I20240502 02:01:23.227156  6404 maintenance_manager.cc:643] P 07cdb213c77c4ffda1f999b66b725399: UndoDeltaBlockGCOp(7f2519b0c6404799ad84d120b45a7ea4) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240502 02:01:23.228757  6515 maintenance_manager.cc:419] P 07cdb213c77c4ffda1f999b66b725399: Scheduling UndoDeltaBlockGCOp(99e74a49b42745888058db67049dc682): 4581 bytes on disk
I20240502 02:01:23.230959  6404 maintenance_manager.cc:643] P 07cdb213c77c4ffda1f999b66b725399: UndoDeltaBlockGCOp(99e74a49b42745888058db67049dc682) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4}
I20240502 02:01:23.237684  6335 consensus_queue.cc:237] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 02:01:23.240864  8268 raft_consensus.cc:1261] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 02:01:23.241559 28469 consensus_queue.cc:1035] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 02:01:23.247673  6469 raft_consensus.cc:1203] T 99e74a49b42745888058db67049dc682 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9]   Dedup: 1.9->[]
I20240502 02:01:23.254096 28674 raft_consensus.cc:2932] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 02:01:23.258484  8268 raft_consensus.cc:2932] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 02:01:23.267630  5922 catalog_manager.cc:5484] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 reported cstate change: config changed from index 7 to 8, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
W20240502 02:01:23.270665  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:01:23.272284  6048 tablet_service.cc:1514] Processing DeleteTablet for tablet afa5561f8c0e4c0fa9bbbd5da61348c7 with delete_type TABLET_DATA_TOMBSTONED (TS 218bc8c1f9ce46059898fc738179adf9 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:58306
I20240502 02:01:23.274618  6335 consensus_queue.cc:237] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 02:01:23.277552  6066 raft_consensus.cc:1261] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 02:01:23.278262 28469 consensus_queue.cc:1035] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 02:01:23.281205  6066 raft_consensus.cc:858] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }.
  Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }: 
  New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
W20240502 02:01:23.281339  6066 raft_consensus.cc:1565] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 9' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }.
  Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }: 
  New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 02:01:23.283459  8268 raft_consensus.cc:1261] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 02:01:23.284994 28674 consensus_queue.cc:1035] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 02:01:23.290872 28469 raft_consensus.cc:2932] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } }
I20240502 02:01:23.294435  8269 raft_consensus.cc:2932] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } }
I20240502 02:01:23.303999  6200 raft_consensus.cc:1203] T 5be45a68cff24a75a03b2fcd4b7851f9 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9]   Dedup: 1.9->[]
I20240502 02:01:23.305233  5921 catalog_manager.cc:5484] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 reported cstate change: config changed from index 8 to 9, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) added. New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } }
I20240502 02:01:23.311393  5907 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet afa5561f8c0e4c0fa9bbbd5da61348c7 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 02:01:23.313411  6251 maintenance_manager.cc:419] P a2c90ee440b44b708bccc2e2d5385cc0: Scheduling UndoDeltaBlockGCOp(5be45a68cff24a75a03b2fcd4b7851f9): 4833 bytes on disk
I20240502 02:01:23.314575  6138 maintenance_manager.cc:643] P a2c90ee440b44b708bccc2e2d5385cc0: UndoDeltaBlockGCOp(5be45a68cff24a75a03b2fcd4b7851f9) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
W20240502 02:01:23.323472  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 02:01:23.324471  6066 raft_consensus.cc:2932] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 02:01:23.329337  6251 maintenance_manager.cc:419] P a2c90ee440b44b708bccc2e2d5385cc0: Scheduling UndoDeltaBlockGCOp(899963a8260a42c4b73fb80eda2ecaa9): 4581 bytes on disk
I20240502 02:01:23.330013  6138 maintenance_manager.cc:643] P a2c90ee440b44b708bccc2e2d5385cc0: UndoDeltaBlockGCOp(899963a8260a42c4b73fb80eda2ecaa9) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20240502 02:01:23.332641  6469 raft_consensus.cc:1203] T 7f2519b0c6404799ad84d120b45a7ea4 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
W20240502 02:01:23.337896  5907 catalog_manager.cc:4804] TS 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227): delete failed for tablet afa5561f8c0e4c0fa9bbbd5da61348c7 because of a CAS failure. No further retry: Illegal state: Request specified cas_config_opid_index_less_or_equal of 7 but the committed config has opid_index of 8
I20240502 02:01:23.341444  6066 raft_consensus.cc:1467] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [term 1 NON_PARTICIPANT]: Allowing update even though not a member of the config
I20240502 02:01:23.342217  6066 raft_consensus.cc:2932] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } }
I20240502 02:01:23.351461 28687 ts_tablet_manager.cc:902] T f309b40931dd444091869119eab33bbc P 60298ffea1dd4c25a38765c2ec9ec9d5: Initiating tablet copy from peer 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867)
I20240502 02:01:23.378283 28687 tablet_copy_client.cc:308] T f309b40931dd444091869119eab33bbc P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.132:36867
I20240502 02:01:23.379401  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:23.387497  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:23.393450  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:23.394356  6200 raft_consensus.cc:1203] T 899963a8260a42c4b73fb80eda2ecaa9 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
W20240502 02:01:23.401969  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:01:23.460494  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:23.463011  6489 tablet_copy_service.cc:140] P 07cdb213c77c4ffda1f999b66b725399: Received BeginTabletCopySession request for tablet f309b40931dd444091869119eab33bbc from peer 60298ffea1dd4c25a38765c2ec9ec9d5 ({username='slave'} at 127.25.109.133:41175)
I20240502 02:01:23.463197  6489 tablet_copy_service.cc:161] P 07cdb213c77c4ffda1f999b66b725399: Beginning new tablet copy session on tablet f309b40931dd444091869119eab33bbc from peer 60298ffea1dd4c25a38765c2ec9ec9d5 at {username='slave'} at 127.25.109.133:41175: session id = 60298ffea1dd4c25a38765c2ec9ec9d5-f309b40931dd444091869119eab33bbc
I20240502 02:01:23.469749  6489 tablet_copy_source_session.cc:205] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 02:01:23.472455 28687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f309b40931dd444091869119eab33bbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:23.490752 28687 tablet_copy_client.cc:783] T f309b40931dd444091869119eab33bbc P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Starting download of 5 data blocks...
I20240502 02:01:23.498766 28687 tablet_copy_client.cc:647] T f309b40931dd444091869119eab33bbc P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Starting download of 1 WAL segments...
I20240502 02:01:23.500763 28687 tablet_copy_client.cc:523] T f309b40931dd444091869119eab33bbc P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:01:23.523262  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:23.528470 28687 tablet_bootstrap.cc:492] T f309b40931dd444091869119eab33bbc P 60298ffea1dd4c25a38765c2ec9ec9d5: Bootstrap starting.
I20240502 02:01:23.566963  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:23.594864 28687 tablet_bootstrap.cc:492] T f309b40931dd444091869119eab33bbc P 60298ffea1dd4c25a38765c2ec9ec9d5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:01:23.595672 28687 tablet_bootstrap.cc:492] T f309b40931dd444091869119eab33bbc P 60298ffea1dd4c25a38765c2ec9ec9d5: Bootstrap complete.
I20240502 02:01:23.596099 28687 ts_tablet_manager.cc:1367] T f309b40931dd444091869119eab33bbc P 60298ffea1dd4c25a38765c2ec9ec9d5: Time spent bootstrapping tablet: real 0.068s	user 0.024s	sys 0.004s
I20240502 02:01:23.596923 28687 raft_consensus.cc:348] T f309b40931dd444091869119eab33bbc P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 02:01:23.597211 28687 raft_consensus.cc:729] T f309b40931dd444091869119eab33bbc P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60298ffea1dd4c25a38765c2ec9ec9d5, State: Initialized, Role: FOLLOWER
I20240502 02:01:23.597534 28687 consensus_queue.cc:260] T f309b40931dd444091869119eab33bbc P 60298ffea1dd4c25a38765c2ec9ec9d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
W20240502 02:01:23.631239  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 02:01:23.636622  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 02:01:23.643726 28687 ts_tablet_manager.cc:1398] T f309b40931dd444091869119eab33bbc P 60298ffea1dd4c25a38765c2ec9ec9d5: Time spent starting tablet: real 0.047s	user 0.016s	sys 0.015s
I20240502 02:01:23.645328  6489 tablet_copy_service.cc:342] P 07cdb213c77c4ffda1f999b66b725399: Request end of tablet copy session 60298ffea1dd4c25a38765c2ec9ec9d5-f309b40931dd444091869119eab33bbc received from {username='slave'} at 127.25.109.133:41175
I20240502 02:01:23.645488  6489 tablet_copy_service.cc:433] P 07cdb213c77c4ffda1f999b66b725399: ending tablet copy session 60298ffea1dd4c25a38765c2ec9ec9d5-f309b40931dd444091869119eab33bbc on tablet f309b40931dd444091869119eab33bbc with peer 60298ffea1dd4c25a38765c2ec9ec9d5
I20240502 02:01:23.698328  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:23.702471  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:01:23.721988  8048 maintenance_manager.cc:419] P 60298ffea1dd4c25a38765c2ec9ec9d5: Scheduling UndoDeltaBlockGCOp(f309b40931dd444091869119eab33bbc): 4581 bytes on disk
I20240502 02:01:23.723140  7937 maintenance_manager.cc:643] P 60298ffea1dd4c25a38765c2ec9ec9d5: UndoDeltaBlockGCOp(f309b40931dd444091869119eab33bbc) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
W20240502 02:01:23.730549  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 02:01:23.732338  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:23.755316  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:23.770321  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:23.780227  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:23.790267  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:23.878540  7999 raft_consensus.cc:1203] T f309b40931dd444091869119eab33bbc P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
W20240502 02:01:23.905824  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240502 02:01:23.918569  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:01:23.984299  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 02:01:24.037792  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:24.048848  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:24.072052  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:24.100934  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:01:24.128177  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 02:01:24.132398  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:24.194757  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:24.213443  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:01:24.247524  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 02:01:24.273829  5907 catalog_manager.cc:4573] TS 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet afa5561f8c0e4c0fa9bbbd5da61348c7: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:01:24.370740  6335 consensus_queue.cc:237] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 02:01:24.383991  8269 raft_consensus.cc:1261] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 02:01:24.384861 28469 consensus_queue.cc:1035] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240502 02:01:24.390071  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 02:01:24.416378  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:24.421146 28852 raft_consensus.cc:2932] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 02:01:24.426290  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:24.435854  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:01:24.440346  5922 catalog_manager.cc:5484] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 reported cstate change: config changed from index 9 to 10, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 02:01:24.444367  8268 raft_consensus.cc:2932] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 02:01:24.454563  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:24.471061  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:01:24.486662  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 02:01:24.492417  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:24.496271  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:24.540114  6048 tablet_service.cc:1514] Processing DeleteTablet for tablet afa5561f8c0e4c0fa9bbbd5da61348c7 with delete_type TABLET_DATA_TOMBSTONED (TS 218bc8c1f9ce46059898fc738179adf9 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:58306
I20240502 02:01:24.544014  6335 consensus_queue.cc:237] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 02:01:24.546327  6066 raft_consensus.cc:1261] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 02:01:24.546401  8268 raft_consensus.cc:1261] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 02:01:24.546980 28854 consensus_queue.cc:1035] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 02:01:24.548372 28855 consensus_queue.cc:1035] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 02:01:24.557240 28852 raft_consensus.cc:2932] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } }
I20240502 02:01:24.562265  6066 raft_consensus.cc:858] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }.
  Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }: 
  New config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
W20240502 02:01:24.562386  6066 raft_consensus.cc:1565] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 11' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }.
  Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }: 
  New config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 02:01:24.564661  8268 raft_consensus.cc:2932] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } }
I20240502 02:01:24.571753  5907 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet afa5561f8c0e4c0fa9bbbd5da61348c7 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 02:01:24.572495  5912 catalog_manager.cc:5484] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 reported cstate change: config changed from index 10 to 11, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) added. New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } }
I20240502 02:01:24.587229  6066 raft_consensus.cc:2932] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
W20240502 02:01:24.593243  5907 catalog_manager.cc:4804] TS 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227): delete failed for tablet afa5561f8c0e4c0fa9bbbd5da61348c7 because of a CAS failure. No further retry: Illegal state: Request specified cas_config_opid_index_less_or_equal of 9 but the committed config has opid_index of 10
I20240502 02:01:24.595520  6066 raft_consensus.cc:1467] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [term 1 NON_PARTICIPANT]: Allowing update even though not a member of the config
I20240502 02:01:24.597604  6066 raft_consensus.cc:2932] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } }
W20240502 02:01:24.610445  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:01:24.665432  6446 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:24.697957  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:24.727777  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:24.731621  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:24.783476  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:24.788265  6469 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0cecd1b14e474dee825fae040ce1b31e"
dest_uuid: "07cdb213c77c4ffda1f999b66b725399"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:40020
I20240502 02:01:24.788527  6469 raft_consensus.cc:595] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Received request to transfer leadership
I20240502 02:01:24.808672  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:24.865056  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 02:01:24.872344 28901 raft_consensus.cc:979] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399: : Instructing follower 94ffd7aaf85b4e17881241b8396fa2f3 to start an election
I20240502 02:01:24.872766 28741 raft_consensus.cc:1067] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Signalling peer 94ffd7aaf85b4e17881241b8396fa2f3 to start an election
I20240502 02:01:24.873827  6335 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0cecd1b14e474dee825fae040ce1b31e"
dest_uuid: "94ffd7aaf85b4e17881241b8396fa2f3"
 from {username='slave'} at 127.25.109.132:33439
I20240502 02:01:24.874120  6335 raft_consensus.cc:482] T 0cecd1b14e474dee825fae040ce1b31e P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 02:01:24.874256  6335 raft_consensus.cc:3037] T 0cecd1b14e474dee825fae040ce1b31e P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 02:01:24.877950  6335 raft_consensus.cc:504] T 0cecd1b14e474dee825fae040ce1b31e P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 02:01:24.880281  6469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cecd1b14e474dee825fae040ce1b31e" candidate_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "07cdb213c77c4ffda1f999b66b725399"
I20240502 02:01:24.880527  6469 raft_consensus.cc:3032] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 02:01:24.880605  6469 raft_consensus.cc:729] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 07cdb213c77c4ffda1f999b66b725399, State: Running, Role: LEADER
I20240502 02:01:24.880882  6469 consensus_queue.cc:260] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 02:01:24.881233  6469 raft_consensus.cc:3037] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Advancing to term 2
I20240502 02:01:24.884897  7999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cecd1b14e474dee825fae040ce1b31e" candidate_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5"
I20240502 02:01:24.885248  7999 raft_consensus.cc:3037] T 0cecd1b14e474dee825fae040ce1b31e P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 02:01:24.890384  7999 raft_consensus.cc:2454] T 0cecd1b14e474dee825fae040ce1b31e P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ffd7aaf85b4e17881241b8396fa2f3 in term 2.
I20240502 02:01:24.890586  6469 raft_consensus.cc:2454] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ffd7aaf85b4e17881241b8396fa2f3 in term 2.
I20240502 02:01:24.891180  6270 leader_election.cc:302] T 0cecd1b14e474dee825fae040ce1b31e P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60298ffea1dd4c25a38765c2ec9ec9d5, 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 
I20240502 02:01:24.891559 28852 raft_consensus.cc:2781] T 0cecd1b14e474dee825fae040ce1b31e P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 02:01:24.891693 28852 raft_consensus.cc:686] T 0cecd1b14e474dee825fae040ce1b31e P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 LEADER]: Becoming Leader. State: Replica: 94ffd7aaf85b4e17881241b8396fa2f3, State: Running, Role: LEADER
I20240502 02:01:24.892277 28852 consensus_queue.cc:237] T 0cecd1b14e474dee825fae040ce1b31e P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 02:01:24.895192  6335 leader_election.cc:288] T 0cecd1b14e474dee825fae040ce1b31e P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 2 election: Requested vote from peers 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867), 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133:43379)
I20240502 02:01:24.906373  5922 catalog_manager.cc:5484] T 0cecd1b14e474dee825fae040ce1b31e P 94ffd7aaf85b4e17881241b8396fa2f3 reported cstate change: term changed from 1 to 2, leader changed from 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) to 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131). New cstate: current_term: 2 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
W20240502 02:01:24.954974  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:01:25.059022  6446 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:25.075100  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:25.106245  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:25.111878  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:25.111951  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:25.115540  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 02:01:25.121711  6314 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 02:01:25.141919  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:01:25.160226  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 02:01:25.280203  6446 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:25.290586  8001 raft_consensus.cc:1261] T 0cecd1b14e474dee825fae040ce1b31e P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 2 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 02:01:25.291709 28852 consensus_queue.cc:1035] T 0cecd1b14e474dee825fae040ce1b31e P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 02:01:25.293102  6469 raft_consensus.cc:1261] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 [term 2 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 02:01:25.293807 28852 consensus_queue.cc:1035] T 0cecd1b14e474dee825fae040ce1b31e P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 02:01:25.310946  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:25.331104  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:25.375285  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 02:01:25.404048  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 02:01:25.419303  6314 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 02:01:25.451133  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240502 02:01:25.453017  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:25.504153  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 02:01:25.512758  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:01:25.522881  6401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240502 02:01:25.586336  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240502 02:01:25.595672  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:01:25.625669  8459 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240502 01:55:44.201637  8463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:44.202200  8463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:44.215312  8463 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:55:47.518293  8463 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 01:55:49.352398  8463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ef6f7a2e933049e5982deec12e9a670a: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 436822a397664c54b8d1b9eb3bc6cff4
W20240502 01:55:49.352743  8463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0053d72412224a79be83b7e007b51934: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:55:49.352933  8463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ef7ebedfbbff4054b074bf7a7622b632: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:55:49.353186  8463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5606c3da105b4b86868c6c51ecdd39aa: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:55:49.353313  8463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c02dbeef0b7241e78a12e4bd7ef79e47: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 436822a397664c54b8d1b9eb3bc6cff4
W20240502 01:55:49.353406  8463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f406d4a60a5e40c485796d37ff4d76bc: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 436822a397664c54b8d1b9eb3bc6cff4
W20240502 01:55:49.353564  8463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1ed4a88787d6401481245c552170a456: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 60298ffea1dd4c25a38765c2ec9ec9d5
W20240502 01:55:49.353650  8463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 63a163be603345428cc84c86b96467f0: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:55:49.353749  8463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fcde569cfdf747c2aaecac7d265b4b1f: could not find any suitable replica to move from server 218bc8c1f9ce46059898fc738179adf9 to server 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:55:49.353842  8463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c2701645b8bc497aaf63e970157bdd80: could not find any suitable replica to move from server 218bc8c1f9ce46059898fc738179adf9 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:55:49.353935  8463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ef6f7a2e933049e5982deec12e9a670a: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:55:49.354035  8463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0053d72412224a79be83b7e007b51934: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:55:49.354144  8463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c02dbeef0b7241e78a12e4bd7ef79e47: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 436822a397664c54b8d1b9eb3bc6cff4
W20240502 01:55:49.354234  8463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5606c3da105b4b86868c6c51ecdd39aa: could not find any suitable replica to move from server 218bc8c1f9ce46059898fc738179adf9 to server 60298ffea1dd4c25a38765c2ec9ec9d5
I20240502 01:55:49.794066  8463 rebalancer_tool.cc:1074] tablet 4486b413e5c14ec6953e4895ee5797cf: '218bc8c1f9ce46059898fc738179adf9' -> '60298ffea1dd4c25a38765c2ec9ec9d5' move scheduled
I20240502 01:55:50.486227  8463 rebalancer_tool.cc:1074] tablet 6984d638991442e3adaed1b24bee95e9: '94ffd7aaf85b4e17881241b8396fa2f3' -> '6d34e50fb5dd4a1db58013d4863b8992' move scheduled
I20240502 01:55:50.943434  8463 rebalancer_tool.cc:1074] tablet 99e74a49b42745888058db67049dc682: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '436822a397664c54b8d1b9eb3bc6cff4' move scheduled
I20240502 01:55:51.456715  8463 rebalancer_tool.cc:1074] tablet c58d6b2401c047cd9af56a66c2e00105: '07cdb213c77c4ffda1f999b66b725399' -> '9db17f84d37944098cbeb9e0d8456efb' move scheduled
I20240502 01:55:51.643198  8463 rebalancer_tool.cc:1084] tablet 3a7698f6bd4349ae832a22c853d76fea: '218bc8c1f9ce46059898fc738179adf9' -> '60298ffea1dd4c25a38765c2ec9ec9d5' move ignored: Illegal state: tablet 3a7698f6bd4349ae832a22c853d76fea: replica 60298ffea1dd4c25a38765c2ec9ec9d5 is already present
I20240502 01:55:51.643414  8463 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:55:53.554688  8463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1ed4a88787d6401481245c552170a456: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:55:53.554958  8463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ef7ebedfbbff4054b074bf7a7622b632: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:55:53.555155  8463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1ed4a88787d6401481245c552170a456: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 436822a397664c54b8d1b9eb3bc6cff4
W20240502 01:55:53.555272  8463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ef7ebedfbbff4054b074bf7a7622b632: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 436822a397664c54b8d1b9eb3bc6cff4
W20240502 01:55:53.555397  8463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f406d4a60a5e40c485796d37ff4d76bc: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:55:53.555570  8463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 63a163be603345428cc84c86b96467f0: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 436822a397664c54b8d1b9eb3bc6cff4
W20240502 01:55:53.555691  8463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c2701645b8bc497aaf63e970157bdd80: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:55:53.555837  8463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c02dbeef0b7241e78a12e4bd7ef79e47: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 60298ffea1dd4c25a38765c2ec9ec9d5
W20240502 01:55:53.555984  8463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5606c3da105b4b86868c6c51ecdd39aa: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 60298ffea1dd4c25a38765c2ec9ec9d5
W20240502 01:55:53.556080  8463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fcde569cfdf747c2aaecac7d265b4b1f: could not find any suitable replica to move from server 218bc8c1f9ce46059898fc738179adf9 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:55:53.556185  8463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1ed4a88787d6401481245c552170a456: could not find any suitable replica to move from server 218bc8c1f9ce46059898fc738179adf9 to server 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:55:53.556279  8463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0053d72412224a79be83b7e007b51934: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 436822a397664c54b8d1b9eb3bc6cff4
W20240502 01:55:53.556391  8463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f406d4a60a5e40c485796d37ff4d76bc: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:55:53.556494  8463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 63a163be603345428cc84c86b96467f0: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 6d34e50fb5dd4a1db58013d4863b8992
I20240502 01:55:53.729763  8463 rebalancer_tool.cc:1074] tablet 9a5f486852554e2fa2d07f7b9d686ced: '218bc8c1f9ce46059898fc738179adf9' -> '' move scheduled
I20240502 01:55:54.258134  8463 rebalancer_tool.cc:1074] tablet f309b40931dd444091869119eab33bbc: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '9db17f84d37944098cbeb9e0d8456efb' move scheduled
I20240502 01:55:54.736687  8463 rebalancer_tool.cc:1084] tablet afa5561f8c0e4c0fa9bbbd5da61348c7: '07cdb213c77c4ffda1f999b66b725399' -> '60298ffea1dd4c25a38765c2ec9ec9d5' move ignored: Illegal state: Request specified cas_config_opid_index of 4 but the committed config has opid_index of 5
I20240502 01:55:54.736829  8463 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:55:56.800441  8463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 63a163be603345428cc84c86b96467f0: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 436822a397664c54b8d1b9eb3bc6cff4
W20240502 01:55:56.801056  8463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fcde569cfdf747c2aaecac7d265b4b1f: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:55:56.801365  8463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ef6f7a2e933049e5982deec12e9a670a: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:55:56.801571  8463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c02dbeef0b7241e78a12e4bd7ef79e47: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:55:56.801733  8463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5606c3da105b4b86868c6c51ecdd39aa: could not find any suitable replica to move from server 218bc8c1f9ce46059898fc738179adf9 to server 436822a397664c54b8d1b9eb3bc6cff4
W20240502 01:55:56.801859  8463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fcde569cfdf747c2aaecac7d265b4b1f: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 436822a397664c54b8d1b9eb3bc6cff4
I20240502 01:55:56.919090  8463 rebalancer_tool.cc:1074] tablet d7922fd756344d14a336ff0c54afdf73: '94ffd7aaf85b4e17881241b8396fa2f3' -> '' move scheduled
I20240502 01:55:57.558348  8463 rebalancer_tool.cc:1074] tablet d8a6db6ae11e4cb8881aee95bdb0e59d: '218bc8c1f9ce46059898fc738179adf9' -> '6d34e50fb5dd4a1db58013d4863b8992' move scheduled
I20240502 01:55:57.768224  8463 rebalancer_tool.cc:1074] tablet 899963a8260a42c4b73fb80eda2ecaa9: '07cdb213c77c4ffda1f999b66b725399' -> '' move scheduled
I20240502 01:55:58.203706  8463 rebalancer_tool.cc:1074] tablet 265a9645e1f141cb88c786b9aef8f90e: '94ffd7aaf85b4e17881241b8396fa2f3' -> '9db17f84d37944098cbeb9e0d8456efb' move scheduled
I20240502 01:55:58.282282  8463 rebalancer_tool.cc:1084] tablet 4f4088f3523f4cd0afdf0de0f447c7af: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '6d34e50fb5dd4a1db58013d4863b8992' move ignored: Illegal state: tablet 4f4088f3523f4cd0afdf0de0f447c7af: replica 6d34e50fb5dd4a1db58013d4863b8992 is already present
I20240502 01:55:58.282446  8463 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:55:59.887524  8463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fcde569cfdf747c2aaecac7d265b4b1f: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:55:59.887742  8463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c02dbeef0b7241e78a12e4bd7ef79e47: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:55:59.888032  8463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5606c3da105b4b86868c6c51ecdd39aa: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 436822a397664c54b8d1b9eb3bc6cff4
W20240502 01:55:59.888175  8463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f406d4a60a5e40c485796d37ff4d76bc: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 60298ffea1dd4c25a38765c2ec9ec9d5
W20240502 01:55:59.888326  8463 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c2701645b8bc497aaf63e970157bdd80: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 436822a397664c54b8d1b9eb3bc6cff4
I20240502 01:55:59.945590  8463 rebalancer_tool.cc:1074] tablet a398da35f18c4df394db82c5bf89aeb1: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '' move scheduled
I20240502 01:56:00.128239  8463 rebalancer_tool.cc:1074] tablet afa5561f8c0e4c0fa9bbbd5da61348c7: '07cdb213c77c4ffda1f999b66b725399' -> '' move scheduled
I20240502 01:56:11.205000  8463 rebalancer_tool.cc:1136] tablet 899963a8260a42c4b73fb80eda2ecaa9: '07cdb213c77c4ffda1f999b66b725399' -> '' move completed: OK
I20240502 01:56:13.662719  8463 rebalancer_tool.cc:1136] tablet 9a5f486852554e2fa2d07f7b9d686ced: '218bc8c1f9ce46059898fc738179adf9' -> '' move completed: OK
W20240502 01:56:22.772246 14987 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 01:56:22.630679 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.134:42517 (local address 127.0.0.1:43622)
0502 01:56:22.630823 (+   144us) negotiation.cc:107] Waiting for socket to connect
0502 01:56:22.630840 (+    17us) client_negotiation.cc:174] Beginning negotiation
0502 01:56:22.630930 (+    90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 01:56:22.770696 (+139766us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":39}
I20240502 01:56:23.082418  8463 rebalancer_tool.cc:1074] tablet 086509f1ee0a4a81b2f8194b50dbe159: '07cdb213c77c4ffda1f999b66b725399' -> '' move scheduled
I20240502 01:56:23.110827  8463 rebalancer_tool.cc:1074] tablet 74eadf4c939c4c6383732cd74c5ec9d6: '218bc8c1f9ce46059898fc738179adf9' -> '' move scheduled
I20240502 01:56:23.197103  8463 rebalancer_tool.cc:1136] tablet a398da35f18c4df394db82c5bf89aeb1: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '' move completed: OK
I20240502 01:56:23.270092  8463 rebalancer_tool.cc:1136] tablet d7922fd756344d14a336ff0c54afdf73: '94ffd7aaf85b4e17881241b8396fa2f3' -> '' move completed: OK
I20240502 01:56:23.425042  8463 rebalancer_tool.cc:1136] tablet 74eadf4c939c4c6383732cd74c5ec9d6: '218bc8c1f9ce46059898fc738179adf9' -> '' move completed: OK
I20240502 01:56:23.568375  8463 rebalancer_tool.cc:1136] tablet 4486b413e5c14ec6953e4895ee5797cf: '218bc8c1f9ce46059898fc738179adf9' -> '60298ffea1dd4c25a38765c2ec9ec9d5' move completed: OK
I20240502 01:56:23.769408  8463 rebalancer_tool.cc:1136] tablet afa5561f8c0e4c0fa9bbbd5da61348c7: '07cdb213c77c4ffda1f999b66b725399' -> '' move completed: OK
I20240502 01:56:23.942723  8463 rebalancer_tool.cc:1136] tablet 99e74a49b42745888058db67049dc682: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '436822a397664c54b8d1b9eb3bc6cff4' move completed: OK
I20240502 01:56:24.119347  8463 rebalancer_tool.cc:1136] tablet 086509f1ee0a4a81b2f8194b50dbe159: '07cdb213c77c4ffda1f999b66b725399' -> '' move completed: OK
I20240502 01:56:24.293540  8463 rebalancer_tool.cc:1136] tablet c58d6b2401c047cd9af56a66c2e00105: '07cdb213c77c4ffda1f999b66b725399' -> '9db17f84d37944098cbeb9e0d8456efb' move completed: OK
I20240502 01:56:24.444104  8463 rebalancer_tool.cc:1136] tablet 265a9645e1f141cb88c786b9aef8f90e: '94ffd7aaf85b4e17881241b8396fa2f3' -> '9db17f84d37944098cbeb9e0d8456efb' move completed: OK
I20240502 01:56:24.548297  8463 rebalancer_tool.cc:1136] tablet d8a6db6ae11e4cb8881aee95bdb0e59d: '218bc8c1f9ce46059898fc738179adf9' -> '6d34e50fb5dd4a1db58013d4863b8992' move completed: OK
I20240502 01:56:24.699921  8463 rebalancer_tool.cc:1136] tablet 6984d638991442e3adaed1b24bee95e9: '94ffd7aaf85b4e17881241b8396fa2f3' -> '6d34e50fb5dd4a1db58013d4863b8992' move completed: OK
I20240502 01:56:26.593832  8463 rebalancer_tool.cc:1074] tablet 02b106ec57b94e44a29947a80a030a23: '218bc8c1f9ce46059898fc738179adf9' -> '' move scheduled
I20240502 01:56:26.675635  8463 rebalancer_tool.cc:1074] tablet 5be45a68cff24a75a03b2fcd4b7851f9: '94ffd7aaf85b4e17881241b8396fa2f3' -> '' move scheduled
I20240502 01:56:26.720558  8463 rebalancer_tool.cc:1074] tablet 4eb91f5df851459b8ecf09cbe34fefa5: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '' move scheduled
I20240502 01:56:26.752132  8463 rebalancer_tool.cc:1074] tablet 55de74ae85e541a19008fd2fc2da3ffd: '218bc8c1f9ce46059898fc738179adf9' -> '' move scheduled
I20240502 01:56:26.861402  8463 rebalancer_tool.cc:1074] tablet 0039dd405cfc4dcea5e7353169e3cd52: '94ffd7aaf85b4e17881241b8396fa2f3' -> '' move scheduled
I20240502 01:56:26.986055  8463 rebalancer_tool.cc:1136] tablet 0039dd405cfc4dcea5e7353169e3cd52: '94ffd7aaf85b4e17881241b8396fa2f3' -> '' move completed: OK
I20240502 01:56:27.158591  8463 rebalancer_tool.cc:1136] tablet 55de74ae85e541a19008fd2fc2da3ffd: '218bc8c1f9ce46059898fc738179adf9' -> '' move completed: OK
I20240502 01:56:27.239193  8463 rebalancer_tool.cc:1136] tablet 4eb91f5df851459b8ecf09cbe34fefa5: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '' move completed: OK
I20240502 01:56:27.278687  8463 rebalancer_tool.cc:1136] tablet 5be45a68cff24a75a03b2fcd4b7851f9: '94ffd7aaf85b4e17881241b8396fa2f3' -> '' move completed: OK
I20240502 01:56:27.323340  8463 rebalancer_tool.cc:1136] tablet 02b106ec57b94e44a29947a80a030a23: '218bc8c1f9ce46059898fc738179adf9' -> '' move completed: OK
I20240502 01:56:29.172704  8463 rebalancer_tool.cc:1074] tablet 89f7cf12d85442749e7d5b259ebca43f: '94ffd7aaf85b4e17881241b8396fa2f3' -> '' move scheduled
I20240502 01:56:29.329284  8463 rebalancer_tool.cc:1074] tablet 866f87ea9f7f48edb667b1b2eb3fa239: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '' move scheduled
I20240502 01:56:29.512604  8463 rebalancer_tool.cc:1136] tablet 866f87ea9f7f48edb667b1b2eb3fa239: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '' move completed: OK
I20240502 01:56:29.594298  8463 rebalancer_tool.cc:1136] tablet 89f7cf12d85442749e7d5b259ebca43f: '94ffd7aaf85b4e17881241b8396fa2f3' -> '' move completed: OK
I20240502 02:01:23.956171  8463 rebalancer_tool.cc:1136] tablet f309b40931dd444091869119eab33bbc: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '9db17f84d37944098cbeb9e0d8456efb' move completed: Incomplete: tablet f309b40931dd444091869119eab33bbc, TS a2c90ee440b44b708bccc2e2d5385cc0 -> TS 9db17f84d37944098cbeb9e0d8456efb move failed, target replica disappeared
Illegal state: tablet server 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): unacceptable health status UNAVAILABLE

I20240502 02:01:25.663528  6335 consensus_queue.cc:237] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 02:01:25.666790  8268 raft_consensus.cc:1261] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 02:01:25.667593 28852 consensus_queue.cc:1035] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 02:01:25.670444 28855 raft_consensus.cc:2932] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 02:01:25.672968  8268 raft_consensus.cc:2932] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 02:01:25.677294  5922 catalog_manager.cc:5484] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 reported cstate change: config changed from index 11 to 12, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 02:01:25.699136  6048 tablet_service.cc:1514] Processing DeleteTablet for tablet afa5561f8c0e4c0fa9bbbd5da61348c7 with delete_type TABLET_DATA_TOMBSTONED (TS 218bc8c1f9ce46059898fc738179adf9 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:58306
I20240502 02:01:25.700138  6335 consensus_queue.cc:237] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 02:01:25.702317  8268 raft_consensus.cc:1261] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 02:01:25.702940  6066 raft_consensus.cc:1261] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 02:01:25.703193 29002 tablet_replica.cc:330] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9: stopping tablet replica
I20240502 02:01:25.703457 29002 raft_consensus.cc:2229] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 02:01:25.703626 28855 consensus_queue.cc:1035] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 02:01:25.703944 29002 raft_consensus.cc:2258] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 02:01:25.705113 29002 ts_tablet_manager.cc:1875] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 02:01:25.705588  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227): Couldn't send request to peer 218bc8c1f9ce46059898fc738179adf9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:01:25.709530 28855 consensus_queue.cc:1035] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 02:01:25.713058 28855 raft_consensus.cc:2932] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } }
I20240502 02:01:25.716300  8268 raft_consensus.cc:2932] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } }
I20240502 02:01:25.723583  5922 catalog_manager.cc:5484] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 reported cstate change: config changed from index 12 to 13, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) added. New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } }
I20240502 02:01:25.725857  5907 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet afa5561f8c0e4c0fa9bbbd5da61348c7 with cas_config_opid_index 12: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 02:01:25.731911 29002 ts_tablet_manager.cc:1888] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20240502 02:01:25.732059 29002 log.cc:1196] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal/wals/afa5561f8c0e4c0fa9bbbd5da61348c7
I20240502 02:01:25.733048  5907 catalog_manager.cc:4830] TS 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227): tablet afa5561f8c0e4c0fa9bbbd5da61348c7 (table rebalance_test_table_2 [id=ef7ebedfbbff4054b074bf7a7622b632]) successfully deleted
I20240502 02:01:25.818626  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:25.819423  6446 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:25.838009  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:25.852818  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:25.854210  6314 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 02:01:25.865177  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:25.988355  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:01:25.998701  6465 consensus_queue.cc:237] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 02:01:26.001015  7999 raft_consensus.cc:1261] T f309b40931dd444091869119eab33bbc P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 02:01:26.001555 28901 consensus_queue.cc:1035] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 02:01:26.004792 28901 raft_consensus.cc:2932] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
W20240502 02:01:26.005441  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 02:01:26.005465  7999 raft_consensus.cc:2932] T f309b40931dd444091869119eab33bbc P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
W20240502 02:01:26.009631  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:01:26.013271  5919 catalog_manager.cc:5484] T f309b40931dd444091869119eab33bbc P 60298ffea1dd4c25a38765c2ec9ec9d5 reported cstate change: config changed from index 7 to 8, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New cstate: current_term: 1 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 02:01:26.021428  6465 consensus_queue.cc:237] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 02:01:26.023183  6200 raft_consensus.cc:1261] T f309b40931dd444091869119eab33bbc P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 02:01:26.023659  7999 raft_consensus.cc:1261] T f309b40931dd444091869119eab33bbc P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 02:01:26.023677 28901 consensus_queue.cc:1035] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 02:01:26.024282 28901 consensus_queue.cc:1035] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 02:01:26.025246  6200 raft_consensus.cc:858] T f309b40931dd444091869119eab33bbc P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }.
  Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }: 
  New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
W20240502 02:01:26.025326  6200 raft_consensus.cc:1565] T f309b40931dd444091869119eab33bbc P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 9' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }.
  Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }: 
  New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 02:01:26.026770  6180 tablet_service.cc:1514] Processing DeleteTablet for tablet f309b40931dd444091869119eab33bbc with delete_type TABLET_DATA_TOMBSTONED (TS a2c90ee440b44b708bccc2e2d5385cc0 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:42126
I20240502 02:01:26.027443 28901 raft_consensus.cc:2932] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } }
I20240502 02:01:26.032069  5905 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet f309b40931dd444091869119eab33bbc with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 02:01:26.032861  7999 raft_consensus.cc:2932] T f309b40931dd444091869119eab33bbc P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } }
I20240502 02:01:26.034514  5919 catalog_manager.cc:5484] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 reported cstate change: config changed from index 8 to 9, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) added. New cstate: current_term: 1 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } }
I20240502 02:01:26.042506  6200 raft_consensus.cc:2932] T f309b40931dd444091869119eab33bbc P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
W20240502 02:01:26.046939  5908 catalog_manager.cc:4804] TS a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197): delete failed for tablet f309b40931dd444091869119eab33bbc because of a CAS failure. No further retry: Illegal state: Request specified cas_config_opid_index_less_or_equal of 7 but the committed config has opid_index of 8
I20240502 02:01:26.048108  6200 raft_consensus.cc:1467] T f309b40931dd444091869119eab33bbc P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 NON_PARTICIPANT]: Allowing update even though not a member of the config
I20240502 02:01:26.048707  6200 raft_consensus.cc:2932] T f309b40931dd444091869119eab33bbc P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } }
I20240502 02:01:26.112314  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:26.119827  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:26.120003  6446 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:26.127210  6314 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 02:01:26.132367  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:26.136312  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 02:01:26.184700  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240502 02:01:26.292762  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 02:01:26.341753  6446 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:26.345614  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:26.347599  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:26.356235  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:01:26.359380  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240502 02:01:26.388971  5907 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.25.109.134:42517, user_credentials={real_user=slave}} blocked reactor thread for 45283.5us
W20240502 02:01:26.404814  5907 catalog_manager.cc:4573] TS 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet afa5561f8c0e4c0fa9bbbd5da61348c7: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:01:26.406180  6314 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 02:01:26.424883  6180 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 02:01:26.528087  6335 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7f2519b0c6404799ad84d120b45a7ea4"
dest_uuid: "94ffd7aaf85b4e17881241b8396fa2f3"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:55476
I20240502 02:01:26.528343  6335 raft_consensus.cc:595] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Received request to transfer leadership
W20240502 02:01:26.537833  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:01:26.563778  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:01:26.679365  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 02:01:26.697830  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 02:01:26.717353  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 02:01:26.730510  6000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 48 similar messages]
I20240502 02:01:26.735591  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:26.752779  6446 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:26.758358  6446 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:26.765801  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:26.773684 29209 ts_tablet_manager.cc:902] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9: Initiating tablet copy from peer 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245)
I20240502 02:01:26.789634  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:26.799008 29218 raft_consensus.cc:979] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3: : Instructing follower a2c90ee440b44b708bccc2e2d5385cc0 to start an election
I20240502 02:01:26.799886 29217 raft_consensus.cc:1067] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Signalling peer a2c90ee440b44b708bccc2e2d5385cc0 to start an election
I20240502 02:01:26.801955  6200 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7f2519b0c6404799ad84d120b45a7ea4"
dest_uuid: "a2c90ee440b44b708bccc2e2d5385cc0"
 from {username='slave'} at 127.25.109.131:59301
I20240502 02:01:26.802259  6200 raft_consensus.cc:482] T 7f2519b0c6404799ad84d120b45a7ea4 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 02:01:26.802377  6200 raft_consensus.cc:3037] T 7f2519b0c6404799ad84d120b45a7ea4 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 02:01:26.806331  6200 raft_consensus.cc:504] T 7f2519b0c6404799ad84d120b45a7ea4 P a2c90ee440b44b708bccc2e2d5385cc0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 02:01:26.811671  6200 leader_election.cc:288] T 7f2519b0c6404799ad84d120b45a7ea4 P a2c90ee440b44b708bccc2e2d5385cc0 [CANDIDATE]: Term 2 election: Requested vote from peers 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245), 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867)
I20240502 02:01:26.811676  6465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f2519b0c6404799ad84d120b45a7ea4" candidate_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "07cdb213c77c4ffda1f999b66b725399"
I20240502 02:01:26.811908  6465 raft_consensus.cc:3037] T 7f2519b0c6404799ad84d120b45a7ea4 P 07cdb213c77c4ffda1f999b66b725399 [term 1 FOLLOWER]: Advancing to term 2
I20240502 02:01:26.813338  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:26.818338  6465 raft_consensus.cc:2454] T 7f2519b0c6404799ad84d120b45a7ea4 P 07cdb213c77c4ffda1f999b66b725399 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c90ee440b44b708bccc2e2d5385cc0 in term 2.
I20240502 02:01:26.818795  6134 leader_election.cc:302] T 7f2519b0c6404799ad84d120b45a7ea4 P a2c90ee440b44b708bccc2e2d5385cc0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07cdb213c77c4ffda1f999b66b725399, a2c90ee440b44b708bccc2e2d5385cc0; no voters: 
I20240502 02:01:26.819130 28734 raft_consensus.cc:2781] T 7f2519b0c6404799ad84d120b45a7ea4 P a2c90ee440b44b708bccc2e2d5385cc0 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 02:01:26.819269 28734 raft_consensus.cc:686] T 7f2519b0c6404799ad84d120b45a7ea4 P a2c90ee440b44b708bccc2e2d5385cc0 [term 2 LEADER]: Becoming Leader. State: Replica: a2c90ee440b44b708bccc2e2d5385cc0, State: Running, Role: LEADER
I20240502 02:01:26.819389  6335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f2519b0c6404799ad84d120b45a7ea4" candidate_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "94ffd7aaf85b4e17881241b8396fa2f3"
I20240502 02:01:26.819622  6335 raft_consensus.cc:3032] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 02:01:26.819550 28734 consensus_queue.cc:237] T 7f2519b0c6404799ad84d120b45a7ea4 P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 02:01:26.819707  6335 raft_consensus.cc:729] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 94ffd7aaf85b4e17881241b8396fa2f3, State: Running, Role: LEADER
I20240502 02:01:26.819983  6335 consensus_queue.cc:260] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 02:01:26.820410  6335 raft_consensus.cc:3037] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 02:01:26.823590  6335 raft_consensus.cc:2454] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c90ee440b44b708bccc2e2d5385cc0 in term 2.
W20240502 02:01:26.832556  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 02:01:26.837226  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 02:01:26.844729 29209 tablet_copy_client.cc:308] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.131:44245
I20240502 02:01:26.847301  6357 tablet_copy_service.cc:140] P 94ffd7aaf85b4e17881241b8396fa2f3: Received BeginTabletCopySession request for tablet afa5561f8c0e4c0fa9bbbd5da61348c7 from peer 218bc8c1f9ce46059898fc738179adf9 ({username='slave'} at 127.25.109.129:40861)
I20240502 02:01:26.847471  6357 tablet_copy_service.cc:161] P 94ffd7aaf85b4e17881241b8396fa2f3: Beginning new tablet copy session on tablet afa5561f8c0e4c0fa9bbbd5da61348c7 from peer 218bc8c1f9ce46059898fc738179adf9 at {username='slave'} at 127.25.109.129:40861: session id = 218bc8c1f9ce46059898fc738179adf9-afa5561f8c0e4c0fa9bbbd5da61348c7
I20240502 02:01:26.851358  6357 tablet_copy_source_session.cc:205] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 02:01:26.853202 29209 ts_tablet_manager.cc:1875] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 02:01:26.856792  5922 catalog_manager.cc:5484] T 7f2519b0c6404799ad84d120b45a7ea4 P a2c90ee440b44b708bccc2e2d5385cc0 reported cstate change: term changed from 1 to 2, leader changed from 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) to a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130). New cstate: current_term: 2 leader_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } }
I20240502 02:01:26.871106  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:26.911561  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:26.924150  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:26.933812 29209 ts_tablet_manager.cc:1888] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.11
I20240502 02:01:26.935336 29209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afa5561f8c0e4c0fa9bbbd5da61348c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:26.937880  6180 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:26.956146  6180 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:26.980674 29209 tablet_copy_client.cc:783] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9: tablet copy: Starting download of 5 data blocks...
I20240502 02:01:26.991518 29209 tablet_copy_client.cc:647] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9: tablet copy: Starting download of 1 WAL segments...
I20240502 02:01:26.993790 29209 tablet_copy_client.cc:523] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:01:27.004817  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:27.013605 29209 tablet_bootstrap.cc:492] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9: Bootstrap starting.
I20240502 02:01:27.062086 29209 tablet_bootstrap.cc:492] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:01:27.063674 29209 tablet_bootstrap.cc:492] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9: Bootstrap complete.
I20240502 02:01:27.064144 29209 ts_tablet_manager.cc:1367] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9: Time spent bootstrapping tablet: real 0.051s	user 0.044s	sys 0.004s
I20240502 02:01:27.065062 29209 raft_consensus.cc:348] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 02:01:27.065395 29209 raft_consensus.cc:729] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218bc8c1f9ce46059898fc738179adf9, State: Initialized, Role: FOLLOWER
I20240502 02:01:27.065750 29209 consensus_queue.cc:260] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 02:01:27.066509 29209 ts_tablet_manager.cc:1398] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 02:01:27.067307  6357 tablet_copy_service.cc:342] P 94ffd7aaf85b4e17881241b8396fa2f3: Request end of tablet copy session 218bc8c1f9ce46059898fc738179adf9-afa5561f8c0e4c0fa9bbbd5da61348c7 received from {username='slave'} at 127.25.109.129:40861
I20240502 02:01:27.067452  6357 tablet_copy_service.cc:433] P 94ffd7aaf85b4e17881241b8396fa2f3: ending tablet copy session 218bc8c1f9ce46059898fc738179adf9-afa5561f8c0e4c0fa9bbbd5da61348c7 on tablet afa5561f8c0e4c0fa9bbbd5da61348c7 with peer 218bc8c1f9ce46059898fc738179adf9
W20240502 02:01:27.134227  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 02:01:27.141306  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:01:27.177068  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:01:27.177883  5907 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 70 similar messages]
I20240502 02:01:27.230060  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:27.231349  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:27.232393  6446 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:27.238677  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:27.248235  6116 maintenance_manager.cc:419] P 218bc8c1f9ce46059898fc738179adf9: Scheduling UndoDeltaBlockGCOp(afa5561f8c0e4c0fa9bbbd5da61348c7): 4581 bytes on disk
I20240502 02:01:27.248962  6002 maintenance_manager.cc:643] P 218bc8c1f9ce46059898fc738179adf9: UndoDeltaBlockGCOp(afa5561f8c0e4c0fa9bbbd5da61348c7) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
I20240502 02:01:27.255847  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:27.272537  6180 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:27.303115  6066 raft_consensus.cc:1203] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.11->[1.12-1.13]   Dedup: 1.13->[]
I20240502 02:01:27.314950  6465 raft_consensus.cc:1261] T 7f2519b0c6404799ad84d120b45a7ea4 P 07cdb213c77c4ffda1f999b66b725399 [term 2 FOLLOWER]: Refusing update from remote peer a2c90ee440b44b708bccc2e2d5385cc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 02:01:27.315512 28734 consensus_queue.cc:1035] T 7f2519b0c6404799ad84d120b45a7ea4 P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 02:01:27.353799  6335 raft_consensus.cc:1261] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Refusing update from remote peer a2c90ee440b44b708bccc2e2d5385cc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 02:01:27.354401 29286 consensus_queue.cc:1035] T 7f2519b0c6404799ad84d120b45a7ea4 P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 02:01:27.421322  6335 consensus_queue.cc:237] T 0cecd1b14e474dee825fae040ce1b31e P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 02:01:27.423342  7999 raft_consensus.cc:1261] T 0cecd1b14e474dee825fae040ce1b31e P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 2 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 02:01:27.423933 28855 consensus_queue.cc:1035] T 0cecd1b14e474dee825fae040ce1b31e P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 02:01:27.426662 29218 raft_consensus.cc:2932] T 0cecd1b14e474dee825fae040ce1b31e P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 02:01:27.427281  7999 raft_consensus.cc:2932] T 0cecd1b14e474dee825fae040ce1b31e P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 02:01:27.436704  5922 catalog_manager.cc:5484] T 0cecd1b14e474dee825fae040ce1b31e P 94ffd7aaf85b4e17881241b8396fa2f3 reported cstate change: config changed from index 7 to 9, VOTER 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132) evicted. New cstate: current_term: 2 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 02:01:27.445580  6446 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cecd1b14e474dee825fae040ce1b31e with delete_type TABLET_DATA_TOMBSTONED (TS 07cdb213c77c4ffda1f999b66b725399 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:39942
I20240502 02:01:27.445626  6335 consensus_queue.cc:237] T 0cecd1b14e474dee825fae040ce1b31e P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
W20240502 02:01:27.448302  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 02:01:27.448891  6270 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227): Couldn't send request to peer 218bc8c1f9ce46059898fc738179adf9. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:01:27.452474 29325 tablet_replica.cc:330] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399: stopping tablet replica
I20240502 02:01:27.452812 29325 raft_consensus.cc:2229] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 02:01:27.453145 29325 raft_consensus.cc:2258] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 02:01:27.454309 29325 ts_tablet_manager.cc:1875] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 02:01:27.459939  8001 raft_consensus.cc:1261] T 0cecd1b14e474dee825fae040ce1b31e P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 2 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 02:01:27.460832 28855 consensus_queue.cc:1035] T 0cecd1b14e474dee825fae040ce1b31e P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 02:01:27.463985 28855 raft_consensus.cc:2932] T 0cecd1b14e474dee825fae040ce1b31e P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } }
I20240502 02:01:27.468664  8001 raft_consensus.cc:2932] T 0cecd1b14e474dee825fae040ce1b31e P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } }
I20240502 02:01:27.474462  5907 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 0cecd1b14e474dee825fae040ce1b31e with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 02:01:27.475797  5922 catalog_manager.cc:5484] T 0cecd1b14e474dee825fae040ce1b31e P 60298ffea1dd4c25a38765c2ec9ec9d5 reported cstate change: config changed from index 9 to 10, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) added. New cstate: current_term: 2 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } }
I20240502 02:01:27.488790 29325 ts_tablet_manager.cc:1888] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20240502 02:01:27.488946 29325 log.cc:1196] T 0cecd1b14e474dee825fae040ce1b31e P 07cdb213c77c4ffda1f999b66b725399: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal/wals/0cecd1b14e474dee825fae040ce1b31e
I20240502 02:01:27.489795  5905 catalog_manager.cc:4830] TS 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867): tablet 0cecd1b14e474dee825fae040ce1b31e (table rebalance_test_table_0 [id=ef6f7a2e933049e5982deec12e9a670a]) successfully deleted
I20240502 02:01:27.539687  6446 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:27.555835  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:27.571816  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:27.572739  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 02:01:27.612936  6136 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 02:01:27.614610  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:01:27.614915  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:01:27.623065  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:27.638135  6180 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 02:01:27.698304  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 02:01:27.795444  6335 consensus_queue.cc:237] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 02:01:27.813557  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:27.818305  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:27.829253  8268 raft_consensus.cc:1261] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240502 02:01:27.829897 28855 consensus_queue.cc:1035] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 02:01:27.836575  6446 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:27.850522 28855 raft_consensus.cc:2932] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 02:01:27.855664  8269 raft_consensus.cc:2932] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 02:01:27.857113  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:27.869383  5916 catalog_manager.cc:5484] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 reported cstate change: config changed from index 13 to 14, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 02:01:27.881440  6048 tablet_service.cc:1514] Processing DeleteTablet for tablet afa5561f8c0e4c0fa9bbbd5da61348c7 with delete_type TABLET_DATA_TOMBSTONED (TS 218bc8c1f9ce46059898fc738179adf9 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:58306
I20240502 02:01:27.893015  6335 consensus_queue.cc:237] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 02:01:27.898198  6067 raft_consensus.cc:1261] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240502 02:01:27.898288  8268 raft_consensus.cc:1261] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240502 02:01:27.899056 29218 consensus_queue.cc:1035] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 02:01:27.900079 29218 consensus_queue.cc:1035] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240502 02:01:27.901230  6067 raft_consensus.cc:858] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }.
  Pending config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }: 
  New config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
W20240502 02:01:27.901362  6067 raft_consensus.cc:1565] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 15' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }.
  Pending config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }: 
  New config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 02:01:27.907838 28855 raft_consensus.cc:2932] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } }
W20240502 02:01:27.909020  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 02:01:27.913262  5907 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet afa5561f8c0e4c0fa9bbbd5da61348c7 with cas_config_opid_index 14: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 02:01:27.916904  5916 catalog_manager.cc:5484] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 reported cstate change: config changed from index 14 to 15, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) added. New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } }
I20240502 02:01:27.930205  6067 raft_consensus.cc:2932] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 02:01:27.932894  8269 raft_consensus.cc:2932] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } }
I20240502 02:01:27.955250  6180 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:27.957016 29404 ts_tablet_manager.cc:902] T 0cecd1b14e474dee825fae040ce1b31e P 218bc8c1f9ce46059898fc738179adf9: Initiating tablet copy from peer 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245)
W20240502 02:01:27.965814  5907 catalog_manager.cc:4804] TS 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227): delete failed for tablet afa5561f8c0e4c0fa9bbbd5da61348c7 because of a CAS failure. No further retry: Illegal state: Request specified cas_config_opid_index_less_or_equal of 13 but the committed config has opid_index of 14
I20240502 02:01:27.968336  6066 raft_consensus.cc:1467] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [term 1 NON_PARTICIPANT]: Allowing update even though not a member of the config
I20240502 02:01:27.970966  6066 raft_consensus.cc:2932] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } }
W20240502 02:01:27.980993  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 02:01:27.984421 29404 tablet_copy_client.cc:308] T 0cecd1b14e474dee825fae040ce1b31e P 218bc8c1f9ce46059898fc738179adf9: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.131:44245
I20240502 02:01:27.985153  6357 tablet_copy_service.cc:140] P 94ffd7aaf85b4e17881241b8396fa2f3: Received BeginTabletCopySession request for tablet 0cecd1b14e474dee825fae040ce1b31e from peer 218bc8c1f9ce46059898fc738179adf9 ({username='slave'} at 127.25.109.129:40861)
I20240502 02:01:27.985316  6357 tablet_copy_service.cc:161] P 94ffd7aaf85b4e17881241b8396fa2f3: Beginning new tablet copy session on tablet 0cecd1b14e474dee825fae040ce1b31e from peer 218bc8c1f9ce46059898fc738179adf9 at {username='slave'} at 127.25.109.129:40861: session id = 218bc8c1f9ce46059898fc738179adf9-0cecd1b14e474dee825fae040ce1b31e
I20240502 02:01:27.988340  6357 tablet_copy_source_session.cc:205] T 0cecd1b14e474dee825fae040ce1b31e P 94ffd7aaf85b4e17881241b8396fa2f3: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 02:01:27.989748 29404 ts_tablet_manager.cc:1875] T 0cecd1b14e474dee825fae040ce1b31e P 218bc8c1f9ce46059898fc738179adf9: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 02:01:27.998064 29404 ts_tablet_manager.cc:1888] T 0cecd1b14e474dee825fae040ce1b31e P 218bc8c1f9ce46059898fc738179adf9: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 02:01:27.998466 29404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cecd1b14e474dee825fae040ce1b31e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:28.001893  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:28.002172 29404 tablet_copy_client.cc:783] T 0cecd1b14e474dee825fae040ce1b31e P 218bc8c1f9ce46059898fc738179adf9: tablet copy: Starting download of 5 data blocks...
I20240502 02:01:28.008266 29404 tablet_copy_client.cc:647] T 0cecd1b14e474dee825fae040ce1b31e P 218bc8c1f9ce46059898fc738179adf9: tablet copy: Starting download of 1 WAL segments...
I20240502 02:01:28.010088 29404 tablet_copy_client.cc:523] T 0cecd1b14e474dee825fae040ce1b31e P 218bc8c1f9ce46059898fc738179adf9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:01:28.014583 29404 tablet_bootstrap.cc:492] T 0cecd1b14e474dee825fae040ce1b31e P 218bc8c1f9ce46059898fc738179adf9: Bootstrap starting.
W20240502 02:01:28.028297  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:01:28.034392 29404 tablet_bootstrap.cc:492] T 0cecd1b14e474dee825fae040ce1b31e P 218bc8c1f9ce46059898fc738179adf9: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:01:28.034752 29404 tablet_bootstrap.cc:492] T 0cecd1b14e474dee825fae040ce1b31e P 218bc8c1f9ce46059898fc738179adf9: Bootstrap complete.
I20240502 02:01:28.035030 29404 ts_tablet_manager.cc:1367] T 0cecd1b14e474dee825fae040ce1b31e P 218bc8c1f9ce46059898fc738179adf9: Time spent bootstrapping tablet: real 0.021s	user 0.016s	sys 0.000s
I20240502 02:01:28.035815 29404 raft_consensus.cc:348] T 0cecd1b14e474dee825fae040ce1b31e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 02:01:28.036055 29404 raft_consensus.cc:729] T 0cecd1b14e474dee825fae040ce1b31e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218bc8c1f9ce46059898fc738179adf9, State: Initialized, Role: FOLLOWER
I20240502 02:01:28.036316 29404 consensus_queue.cc:260] T 0cecd1b14e474dee825fae040ce1b31e P 218bc8c1f9ce46059898fc738179adf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 02:01:28.036819 29404 ts_tablet_manager.cc:1398] T 0cecd1b14e474dee825fae040ce1b31e P 218bc8c1f9ce46059898fc738179adf9: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240502 02:01:28.037933  6357 tablet_copy_service.cc:342] P 94ffd7aaf85b4e17881241b8396fa2f3: Request end of tablet copy session 218bc8c1f9ce46059898fc738179adf9-0cecd1b14e474dee825fae040ce1b31e received from {username='slave'} at 127.25.109.129:40861
I20240502 02:01:28.038066  6357 tablet_copy_service.cc:433] P 94ffd7aaf85b4e17881241b8396fa2f3: ending tablet copy session 218bc8c1f9ce46059898fc738179adf9-0cecd1b14e474dee825fae040ce1b31e on tablet 0cecd1b14e474dee825fae040ce1b31e with peer 218bc8c1f9ce46059898fc738179adf9
W20240502 02:01:28.046324  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:01:28.087028  6465 consensus_queue.cc:237] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 02:01:28.089332  8001 raft_consensus.cc:1261] T f309b40931dd444091869119eab33bbc P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 02:01:28.089956 28901 consensus_queue.cc:1035] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 02:01:28.092733 29441 raft_consensus.cc:2932] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 02:01:28.093365  8001 raft_consensus.cc:2932] T f309b40931dd444091869119eab33bbc P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 02:01:28.099808  5916 catalog_manager.cc:5484] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 reported cstate change: config changed from index 9 to 10, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) evicted. New cstate: current_term: 1 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 02:01:28.104053  6180 tablet_service.cc:1514] Processing DeleteTablet for tablet f309b40931dd444091869119eab33bbc with delete_type TABLET_DATA_TOMBSTONED (TS a2c90ee440b44b708bccc2e2d5385cc0 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:42126
I20240502 02:01:28.105084  6465 consensus_queue.cc:237] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 02:01:28.107205  8001 raft_consensus.cc:1261] T f309b40931dd444091869119eab33bbc P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Refusing update from remote peer 07cdb213c77c4ffda1f999b66b725399: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240502 02:01:28.107573  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227): Couldn't send request to peer 218bc8c1f9ce46059898fc738179adf9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f309b40931dd444091869119eab33bbc. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:01:28.107916 29441 consensus_queue.cc:1035] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 02:01:28.111476 28901 raft_consensus.cc:2932] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } }
I20240502 02:01:28.115343 29445 tablet_replica.cc:330] T f309b40931dd444091869119eab33bbc P a2c90ee440b44b708bccc2e2d5385cc0: stopping tablet replica
I20240502 02:01:28.116096 29445 raft_consensus.cc:2229] T f309b40931dd444091869119eab33bbc P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 02:01:28.116448 29445 raft_consensus.cc:2258] T f309b40931dd444091869119eab33bbc P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 02:01:28.117580 29445 ts_tablet_manager.cc:1875] T f309b40931dd444091869119eab33bbc P a2c90ee440b44b708bccc2e2d5385cc0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 02:01:28.119115  8001 raft_consensus.cc:2932] T f309b40931dd444091869119eab33bbc P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } }
I20240502 02:01:28.122993  5905 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet f309b40931dd444091869119eab33bbc with cas_config_opid_index 10: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 02:01:28.125007  5916 catalog_manager.cc:5484] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 reported cstate change: config changed from index 10 to 11, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) added. New cstate: current_term: 1 leader_uuid: "07cdb213c77c4ffda1f999b66b725399" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } }
I20240502 02:01:28.134655 29445 ts_tablet_manager.cc:1888] T f309b40931dd444091869119eab33bbc P a2c90ee440b44b708bccc2e2d5385cc0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 02:01:28.134807 29445 log.cc:1196] T f309b40931dd444091869119eab33bbc P a2c90ee440b44b708bccc2e2d5385cc0: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal/wals/f309b40931dd444091869119eab33bbc
I20240502 02:01:28.135485  5908 catalog_manager.cc:4830] TS a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197): tablet f309b40931dd444091869119eab33bbc (table rebalance_test_table_9 [id=f406d4a60a5e40c485796d37ff4d76bc]) successfully deleted
W20240502 02:01:28.139015  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240502 02:01:28.148561  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
I20240502 02:01:28.248663  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:28.253742  6446 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:28.255106  6116 maintenance_manager.cc:419] P 218bc8c1f9ce46059898fc738179adf9: Scheduling UndoDeltaBlockGCOp(0cecd1b14e474dee825fae040ce1b31e): 4834 bytes on disk
I20240502 02:01:28.255766  6002 maintenance_manager.cc:643] P 218bc8c1f9ce46059898fc738179adf9: UndoDeltaBlockGCOp(0cecd1b14e474dee825fae040ce1b31e) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
I20240502 02:01:28.256547  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:28.258452  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:28.268544  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:28.289078  6180 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:28.303926  6066 raft_consensus.cc:1203] T 0cecd1b14e474dee825fae040ce1b31e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10]   Dedup: 2.10->[]
W20240502 02:01:28.330271  6269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240502 02:01:28.405618  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:01:28.432968  6335 consensus_queue.cc:237] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 1.15, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 02:01:28.434772  8268 raft_consensus.cc:1261] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20240502 02:01:28.435348 29218 consensus_queue.cc:1035] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240502 02:01:28.437834 29218 raft_consensus.cc:2932] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Committing config change with OpId 1.16: config changed from index 15 to 16, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 02:01:28.440390  8268 raft_consensus.cc:2932] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 02:01:28.446110  5913 catalog_manager.cc:5484] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 reported cstate change: config changed from index 15 to 16, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 02:01:28.452459  6048 tablet_service.cc:1514] Processing DeleteTablet for tablet afa5561f8c0e4c0fa9bbbd5da61348c7 with delete_type TABLET_DATA_TOMBSTONED (TS 218bc8c1f9ce46059898fc738179adf9 not found in new config with opid_index 16) from {username='slave'} at 127.0.0.1:58306
I20240502 02:01:28.453192 29403 tablet_replica.cc:330] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9: stopping tablet replica
I20240502 02:01:28.453455 29403 raft_consensus.cc:2229] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 02:01:28.453758 29403 raft_consensus.cc:2258] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 02:01:28.453980  6335 consensus_queue.cc:237] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 1.16, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
W20240502 02:01:28.456290  6270 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227): Couldn't send request to peer 218bc8c1f9ce46059898fc738179adf9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:01:28.456328  8268 raft_consensus.cc:1261] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20240502 02:01:28.457665 29406 consensus_queue.cc:1035] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20240502 02:01:28.460420 29406 raft_consensus.cc:2932] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Committing config change with OpId 1.17: config changed from index 16 to 17, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) added. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } }
I20240502 02:01:28.461561 29403 ts_tablet_manager.cc:1875] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 02:01:28.464171  5907 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet afa5561f8c0e4c0fa9bbbd5da61348c7 with cas_config_opid_index 16: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 02:01:28.465082  8268 raft_consensus.cc:2932] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index 16 to 17, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) added. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } }
I20240502 02:01:28.469175 29403 ts_tablet_manager.cc:1888] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.15
I20240502 02:01:28.469324 29403 log.cc:1196] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal/wals/afa5561f8c0e4c0fa9bbbd5da61348c7
I20240502 02:01:28.470106  5907 catalog_manager.cc:4830] TS 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227): tablet afa5561f8c0e4c0fa9bbbd5da61348c7 (table rebalance_test_table_2 [id=ef7ebedfbbff4054b074bf7a7622b632]) successfully deleted
I20240502 02:01:28.470504  5913 catalog_manager.cc:5484] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 reported cstate change: config changed from index 16 to 17, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) added. New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } }
I20240502 02:01:28.514197  6446 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:28.528633  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:28.531157  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:28.540374  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:28.544257  6180 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:28.549645  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 02:01:28.634455  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:01:28.649114 29544 ts_tablet_manager.cc:902] T f309b40931dd444091869119eab33bbc P 218bc8c1f9ce46059898fc738179adf9: Initiating tablet copy from peer 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867)
I20240502 02:01:28.657642 29544 tablet_copy_client.cc:308] T f309b40931dd444091869119eab33bbc P 218bc8c1f9ce46059898fc738179adf9: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.132:36867
I20240502 02:01:28.658733  6489 tablet_copy_service.cc:140] P 07cdb213c77c4ffda1f999b66b725399: Received BeginTabletCopySession request for tablet f309b40931dd444091869119eab33bbc from peer 218bc8c1f9ce46059898fc738179adf9 ({username='slave'} at 127.25.109.129:53907)
I20240502 02:01:28.658915  6489 tablet_copy_service.cc:161] P 07cdb213c77c4ffda1f999b66b725399: Beginning new tablet copy session on tablet f309b40931dd444091869119eab33bbc from peer 218bc8c1f9ce46059898fc738179adf9 at {username='slave'} at 127.25.109.129:53907: session id = 218bc8c1f9ce46059898fc738179adf9-f309b40931dd444091869119eab33bbc
I20240502 02:01:28.662391  6489 tablet_copy_source_session.cc:205] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 02:01:28.663933 29544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f309b40931dd444091869119eab33bbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:28.675556 29544 tablet_copy_client.cc:783] T f309b40931dd444091869119eab33bbc P 218bc8c1f9ce46059898fc738179adf9: tablet copy: Starting download of 5 data blocks...
I20240502 02:01:28.683158 29544 tablet_copy_client.cc:647] T f309b40931dd444091869119eab33bbc P 218bc8c1f9ce46059898fc738179adf9: tablet copy: Starting download of 1 WAL segments...
I20240502 02:01:28.684994 29544 tablet_copy_client.cc:523] T f309b40931dd444091869119eab33bbc P 218bc8c1f9ce46059898fc738179adf9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240502 02:01:28.687500  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
I20240502 02:01:28.689985 29544 tablet_bootstrap.cc:492] T f309b40931dd444091869119eab33bbc P 218bc8c1f9ce46059898fc738179adf9: Bootstrap starting.
I20240502 02:01:28.723886 29544 tablet_bootstrap.cc:492] T f309b40931dd444091869119eab33bbc P 218bc8c1f9ce46059898fc738179adf9: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:01:28.724344 29544 tablet_bootstrap.cc:492] T f309b40931dd444091869119eab33bbc P 218bc8c1f9ce46059898fc738179adf9: Bootstrap complete.
I20240502 02:01:28.724642 29544 ts_tablet_manager.cc:1367] T f309b40931dd444091869119eab33bbc P 218bc8c1f9ce46059898fc738179adf9: Time spent bootstrapping tablet: real 0.035s	user 0.022s	sys 0.012s
I20240502 02:01:28.725287 29544 raft_consensus.cc:348] T f309b40931dd444091869119eab33bbc P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 02:01:28.725492 29544 raft_consensus.cc:729] T f309b40931dd444091869119eab33bbc P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218bc8c1f9ce46059898fc738179adf9, State: Initialized, Role: FOLLOWER
I20240502 02:01:28.725811 29544 consensus_queue.cc:260] T f309b40931dd444091869119eab33bbc P 218bc8c1f9ce46059898fc738179adf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 02:01:28.726398 29544 ts_tablet_manager.cc:1398] T f309b40931dd444091869119eab33bbc P 218bc8c1f9ce46059898fc738179adf9: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 02:01:28.727177  6489 tablet_copy_service.cc:342] P 07cdb213c77c4ffda1f999b66b725399: Request end of tablet copy session 218bc8c1f9ce46059898fc738179adf9-f309b40931dd444091869119eab33bbc received from {username='slave'} at 127.25.109.129:53907
I20240502 02:01:28.727308  6489 tablet_copy_service.cc:433] P 07cdb213c77c4ffda1f999b66b725399: ending tablet copy session 218bc8c1f9ce46059898fc738179adf9-f309b40931dd444091869119eab33bbc on tablet f309b40931dd444091869119eab33bbc with peer 218bc8c1f9ce46059898fc738179adf9
I20240502 02:01:28.758952  6116 maintenance_manager.cc:419] P 218bc8c1f9ce46059898fc738179adf9: Scheduling UndoDeltaBlockGCOp(f309b40931dd444091869119eab33bbc): 4581 bytes on disk
I20240502 02:01:28.759656  6002 maintenance_manager.cc:643] P 218bc8c1f9ce46059898fc738179adf9: UndoDeltaBlockGCOp(f309b40931dd444091869119eab33bbc) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
I20240502 02:01:28.761312  6200 consensus_queue.cc:237] T 7f2519b0c6404799ad84d120b45a7ea4 P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 02:01:28.764045  6465 raft_consensus.cc:1261] T 7f2519b0c6404799ad84d120b45a7ea4 P 07cdb213c77c4ffda1f999b66b725399 [term 2 FOLLOWER]: Refusing update from remote peer a2c90ee440b44b708bccc2e2d5385cc0: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 02:01:28.764739 29286 consensus_queue.cc:1035] T 7f2519b0c6404799ad84d120b45a7ea4 P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 02:01:28.772679  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 02:01:28.772886 29590 raft_consensus.cc:2932] T 7f2519b0c6404799ad84d120b45a7ea4 P a2c90ee440b44b708bccc2e2d5385cc0 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } }
I20240502 02:01:28.774473  6465 raft_consensus.cc:2932] T 7f2519b0c6404799ad84d120b45a7ea4 P 07cdb213c77c4ffda1f999b66b725399 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } }
I20240502 02:01:28.779539  5922 catalog_manager.cc:5484] T 7f2519b0c6404799ad84d120b45a7ea4 P a2c90ee440b44b708bccc2e2d5385cc0 reported cstate change: config changed from index 7 to 9, VOTER 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131) evicted. New cstate: current_term: 2 leader_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } }
I20240502 02:01:28.786736  6314 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f2519b0c6404799ad84d120b45a7ea4 with delete_type TABLET_DATA_TOMBSTONED (TS 94ffd7aaf85b4e17881241b8396fa2f3 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:59958
I20240502 02:01:28.787348  6200 consensus_queue.cc:237] T 7f2519b0c6404799ad84d120b45a7ea4 P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 02:01:28.787953 29594 tablet_replica.cc:330] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3: stopping tablet replica
I20240502 02:01:28.788250 29594 raft_consensus.cc:2229] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 02:01:28.788491 29594 raft_consensus.cc:2258] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 02:01:28.790627 29594 ts_tablet_manager.cc:1875] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 02:01:28.798077  6465 raft_consensus.cc:1261] T 7f2519b0c6404799ad84d120b45a7ea4 P 07cdb213c77c4ffda1f999b66b725399 [term 2 FOLLOWER]: Refusing update from remote peer a2c90ee440b44b708bccc2e2d5385cc0: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 02:01:28.798769 29590 consensus_queue.cc:1035] T 7f2519b0c6404799ad84d120b45a7ea4 P a2c90ee440b44b708bccc2e2d5385cc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240502 02:01:28.799929  6136 consensus_peers.cc:485] T 7f2519b0c6404799ad84d120b45a7ea4 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133:43379): Couldn't send request to peer 60298ffea1dd4c25a38765c2ec9ec9d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7f2519b0c6404799ad84d120b45a7ea4. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:01:28.803560 29286 raft_consensus.cc:2932] T 7f2519b0c6404799ad84d120b45a7ea4 P a2c90ee440b44b708bccc2e2d5385cc0 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 02:01:28.808277  6469 raft_consensus.cc:2932] T 7f2519b0c6404799ad84d120b45a7ea4 P 07cdb213c77c4ffda1f999b66b725399 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
W20240502 02:01:28.826987  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:01:28.829666  5908 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 7f2519b0c6404799ad84d120b45a7ea4 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 02:01:28.830745  5922 catalog_manager.cc:5484] T 7f2519b0c6404799ad84d120b45a7ea4 P 07cdb213c77c4ffda1f999b66b725399 reported cstate change: config changed from index 9 to 10, VOTER 60298ffea1dd4c25a38765c2ec9ec9d5 (127.25.109.133) added. New cstate: current_term: 2 leader_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } } }
I20240502 02:01:28.846334 29594 ts_tablet_manager.cc:1888] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20240502 02:01:28.846503 29594 log.cc:1196] T 7f2519b0c6404799ad84d120b45a7ea4 P 94ffd7aaf85b4e17881241b8396fa2f3: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/wals/7f2519b0c6404799ad84d120b45a7ea4
I20240502 02:01:28.847429  5907 catalog_manager.cc:4830] TS 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245): tablet 7f2519b0c6404799ad84d120b45a7ea4 (table rebalance_test_table_3 [id=63a163be603345428cc84c86b96467f0]) successfully deleted
W20240502 02:01:28.890130  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 02:01:28.891100  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
I20240502 02:01:28.902441  6446 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:28.926367  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:28.937338  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:28.942107  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:28.958798  6180 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:28.980981  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:29.015519  6066 raft_consensus.cc:1203] T f309b40931dd444091869119eab33bbc P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11]   Dedup: 1.11->[]
W20240502 02:01:29.059897  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 02:01:29.149025  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:01:29.152755  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 02:01:29.174682  5999 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 02:01:29.186573  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 02:01:29.230728  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:29.240654  6446 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:29.246212  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:29.255751  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:29.258586  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:29.269100 29668 ts_tablet_manager.cc:902] T 7f2519b0c6404799ad84d120b45a7ea4 P 60298ffea1dd4c25a38765c2ec9ec9d5: Initiating tablet copy from peer a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197)
I20240502 02:01:29.278216  6180 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:29.291725 29668 tablet_copy_client.cc:308] T 7f2519b0c6404799ad84d120b45a7ea4 P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.130:38197
I20240502 02:01:29.311350  6224 tablet_copy_service.cc:140] P a2c90ee440b44b708bccc2e2d5385cc0: Received BeginTabletCopySession request for tablet 7f2519b0c6404799ad84d120b45a7ea4 from peer 60298ffea1dd4c25a38765c2ec9ec9d5 ({username='slave'} at 127.25.109.133:36303)
I20240502 02:01:29.311558  6224 tablet_copy_service.cc:161] P a2c90ee440b44b708bccc2e2d5385cc0: Beginning new tablet copy session on tablet 7f2519b0c6404799ad84d120b45a7ea4 from peer 60298ffea1dd4c25a38765c2ec9ec9d5 at {username='slave'} at 127.25.109.133:36303: session id = 60298ffea1dd4c25a38765c2ec9ec9d5-7f2519b0c6404799ad84d120b45a7ea4
I20240502 02:01:29.316077  6224 tablet_copy_source_session.cc:205] T 7f2519b0c6404799ad84d120b45a7ea4 P a2c90ee440b44b708bccc2e2d5385cc0: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 02:01:29.317972 29668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f2519b0c6404799ad84d120b45a7ea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:29.326412 29668 tablet_copy_client.cc:783] T 7f2519b0c6404799ad84d120b45a7ea4 P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Starting download of 5 data blocks...
I20240502 02:01:29.334455 29668 tablet_copy_client.cc:647] T 7f2519b0c6404799ad84d120b45a7ea4 P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Starting download of 1 WAL segments...
I20240502 02:01:29.336270 29668 tablet_copy_client.cc:523] T 7f2519b0c6404799ad84d120b45a7ea4 P 60298ffea1dd4c25a38765c2ec9ec9d5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:01:29.340644 29668 tablet_bootstrap.cc:492] T 7f2519b0c6404799ad84d120b45a7ea4 P 60298ffea1dd4c25a38765c2ec9ec9d5: Bootstrap starting.
I20240502 02:01:29.360810 29545 raft_consensus.cc:482] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 02:01:29.361073 29545 raft_consensus.cc:504] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 02:01:29.361972 29545 leader_election.cc:288] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 02:01:29.365234  5999 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:01:29.377595  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:01:29.382691  6000 leader_election.cc:334] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:01:29.382856  6000 leader_election.cc:302] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 218bc8c1f9ce46059898fc738179adf9; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 02:01:29.383145 29545 raft_consensus.cc:2726] T 265a9645e1f141cb88c786b9aef8f90e P 218bc8c1f9ce46059898fc738179adf9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 02:01:29.423671 29683 ts_tablet_manager.cc:902] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9: Initiating tablet copy from peer 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245)
I20240502 02:01:29.434510 29668 tablet_bootstrap.cc:492] T 7f2519b0c6404799ad84d120b45a7ea4 P 60298ffea1dd4c25a38765c2ec9ec9d5: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:01:29.435092 29668 tablet_bootstrap.cc:492] T 7f2519b0c6404799ad84d120b45a7ea4 P 60298ffea1dd4c25a38765c2ec9ec9d5: Bootstrap complete.
I20240502 02:01:29.435392 29668 ts_tablet_manager.cc:1367] T 7f2519b0c6404799ad84d120b45a7ea4 P 60298ffea1dd4c25a38765c2ec9ec9d5: Time spent bootstrapping tablet: real 0.095s	user 0.051s	sys 0.007s
I20240502 02:01:29.436239 29668 raft_consensus.cc:348] T 7f2519b0c6404799ad84d120b45a7ea4 P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 02:01:29.436470 29668 raft_consensus.cc:729] T 7f2519b0c6404799ad84d120b45a7ea4 P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60298ffea1dd4c25a38765c2ec9ec9d5, State: Initialized, Role: FOLLOWER
I20240502 02:01:29.436759 29668 consensus_queue.cc:260] T 7f2519b0c6404799ad84d120b45a7ea4 P 60298ffea1dd4c25a38765c2ec9ec9d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } } peers { permanent_uuid: "60298ffea1dd4c25a38765c2ec9ec9d5" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 43379 } }
I20240502 02:01:29.450736 29683 tablet_copy_client.cc:308] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.131:44245
I20240502 02:01:29.454514  6357 tablet_copy_service.cc:140] P 94ffd7aaf85b4e17881241b8396fa2f3: Received BeginTabletCopySession request for tablet afa5561f8c0e4c0fa9bbbd5da61348c7 from peer 218bc8c1f9ce46059898fc738179adf9 ({username='slave'} at 127.25.109.129:40861)
I20240502 02:01:29.454699  6357 tablet_copy_service.cc:161] P 94ffd7aaf85b4e17881241b8396fa2f3: Beginning new tablet copy session on tablet afa5561f8c0e4c0fa9bbbd5da61348c7 from peer 218bc8c1f9ce46059898fc738179adf9 at {username='slave'} at 127.25.109.129:40861: session id = 218bc8c1f9ce46059898fc738179adf9-afa5561f8c0e4c0fa9bbbd5da61348c7
I20240502 02:01:29.458346  6357 tablet_copy_source_session.cc:205] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 02:01:29.459758 29668 ts_tablet_manager.cc:1398] T 7f2519b0c6404799ad84d120b45a7ea4 P 60298ffea1dd4c25a38765c2ec9ec9d5: Time spent starting tablet: real 0.024s	user 0.019s	sys 0.004s
I20240502 02:01:29.459771 29683 ts_tablet_manager.cc:1875] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 02:01:29.460613  6224 tablet_copy_service.cc:342] P a2c90ee440b44b708bccc2e2d5385cc0: Request end of tablet copy session 60298ffea1dd4c25a38765c2ec9ec9d5-7f2519b0c6404799ad84d120b45a7ea4 received from {username='slave'} at 127.25.109.133:36303
I20240502 02:01:29.460762  6224 tablet_copy_service.cc:433] P a2c90ee440b44b708bccc2e2d5385cc0: ending tablet copy session 60298ffea1dd4c25a38765c2ec9ec9d5-7f2519b0c6404799ad84d120b45a7ea4 on tablet 7f2519b0c6404799ad84d120b45a7ea4 with peer 60298ffea1dd4c25a38765c2ec9ec9d5
I20240502 02:01:29.471295 29683 ts_tablet_manager.cc:1888] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.15
I20240502 02:01:29.471635 29683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afa5561f8c0e4c0fa9bbbd5da61348c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:29.479008 29683 tablet_copy_client.cc:783] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9: tablet copy: Starting download of 5 data blocks...
W20240502 02:01:29.479472  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 02:01:29.485746 29683 tablet_copy_client.cc:647] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9: tablet copy: Starting download of 1 WAL segments...
I20240502 02:01:29.487358 29683 tablet_copy_client.cc:523] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:01:29.490653 29683 tablet_bootstrap.cc:492] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9: Bootstrap starting.
I20240502 02:01:29.499590  8048 maintenance_manager.cc:419] P 60298ffea1dd4c25a38765c2ec9ec9d5: Scheduling UndoDeltaBlockGCOp(7f2519b0c6404799ad84d120b45a7ea4): 4921 bytes on disk
I20240502 02:01:29.500525  7937 maintenance_manager.cc:643] P 60298ffea1dd4c25a38765c2ec9ec9d5: UndoDeltaBlockGCOp(7f2519b0c6404799ad84d120b45a7ea4) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
W20240502 02:01:29.516049  6135 consensus_peers.cc:485] T 73ed91df7ff04846991c95b30a86ea7e P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 02:01:29.520170 29683 tablet_bootstrap.cc:492] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9: Bootstrap replayed 1/1 log segments. Stats: ops{read=17 overwritten=0 applied=17 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:01:29.520556 29683 tablet_bootstrap.cc:492] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9: Bootstrap complete.
I20240502 02:01:29.520821 29683 ts_tablet_manager.cc:1367] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9: Time spent bootstrapping tablet: real 0.030s	user 0.023s	sys 0.008s
I20240502 02:01:29.521579 29683 raft_consensus.cc:348] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 02:01:29.521770 29683 raft_consensus.cc:729] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218bc8c1f9ce46059898fc738179adf9, State: Initialized, Role: FOLLOWER
I20240502 02:01:29.522022 29683 consensus_queue.cc:260] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 17, Last appended: 1.17, Last appended by leader: 17, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } }
I20240502 02:01:29.522629 29683 ts_tablet_manager.cc:1398] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 02:01:29.523726  6357 tablet_copy_service.cc:342] P 94ffd7aaf85b4e17881241b8396fa2f3: Request end of tablet copy session 218bc8c1f9ce46059898fc738179adf9-afa5561f8c0e4c0fa9bbbd5da61348c7 received from {username='slave'} at 127.25.109.129:40861
I20240502 02:01:29.523895  6357 tablet_copy_service.cc:433] P 94ffd7aaf85b4e17881241b8396fa2f3: ending tablet copy session 218bc8c1f9ce46059898fc738179adf9-afa5561f8c0e4c0fa9bbbd5da61348c7 on tablet afa5561f8c0e4c0fa9bbbd5da61348c7 with peer 218bc8c1f9ce46059898fc738179adf9
W20240502 02:01:29.623389  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:01:29.641726  6446 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:29.670719  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:29.675381  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:29.716527  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:29.724543  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:29.733678  6180 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:29.745486  8001 raft_consensus.cc:1203] T 7f2519b0c6404799ad84d120b45a7ea4 P 60298ffea1dd4c25a38765c2ec9ec9d5 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10]   Dedup: 2.10->[]
I20240502 02:01:29.765034  6116 maintenance_manager.cc:419] P 218bc8c1f9ce46059898fc738179adf9: Scheduling UndoDeltaBlockGCOp(afa5561f8c0e4c0fa9bbbd5da61348c7): 4581 bytes on disk
I20240502 02:01:29.765784  6002 maintenance_manager.cc:643] P 218bc8c1f9ce46059898fc738179adf9: UndoDeltaBlockGCOp(afa5561f8c0e4c0fa9bbbd5da61348c7) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
W20240502 02:01:29.774454  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:01:29.783430  6066 raft_consensus.cc:1203] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.16->[1.17-1.17]   Dedup: 1.17->[]
W20240502 02:01:29.959008  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 02:01:30.022380  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 02:01:30.039005  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:01:30.074699  8456 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240502 01:55:44.221165  8464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:44.221578  8464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:44.234583  8464 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:55:47.445298  8464 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 01:55:49.289870  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 63a163be603345428cc84c86b96467f0: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:55:49.290119  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ef7ebedfbbff4054b074bf7a7622b632: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:55:49.290339  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ef6f7a2e933049e5982deec12e9a670a: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 436822a397664c54b8d1b9eb3bc6cff4
W20240502 01:55:49.290484  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1ed4a88787d6401481245c552170a456: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 60298ffea1dd4c25a38765c2ec9ec9d5
W20240502 01:55:49.290589  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fcde569cfdf747c2aaecac7d265b4b1f: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 436822a397664c54b8d1b9eb3bc6cff4
W20240502 01:55:49.290730  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0053d72412224a79be83b7e007b51934: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:55:49.290882  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5606c3da105b4b86868c6c51ecdd39aa: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 60298ffea1dd4c25a38765c2ec9ec9d5
W20240502 01:55:49.291018  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c02dbeef0b7241e78a12e4bd7ef79e47: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:55:49.291115  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f406d4a60a5e40c485796d37ff4d76bc: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:55:49.291229  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c2701645b8bc497aaf63e970157bdd80: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:55:49.291337  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fcde569cfdf747c2aaecac7d265b4b1f: could not find any suitable replica to move from server 218bc8c1f9ce46059898fc738179adf9 to server 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:55:49.291468  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ef6f7a2e933049e5982deec12e9a670a: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:55:49.291549  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0053d72412224a79be83b7e007b51934: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 60298ffea1dd4c25a38765c2ec9ec9d5
W20240502 01:55:49.291651  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c2701645b8bc497aaf63e970157bdd80: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 436822a397664c54b8d1b9eb3bc6cff4
W20240502 01:55:49.291749  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ef7ebedfbbff4054b074bf7a7622b632: could not find any suitable replica to move from server 218bc8c1f9ce46059898fc738179adf9 to server 6d34e50fb5dd4a1db58013d4863b8992
I20240502 01:55:49.590768  8464 rebalancer_tool.cc:1074] tablet 9a5f486852554e2fa2d07f7b9d686ced: '218bc8c1f9ce46059898fc738179adf9' -> '6d34e50fb5dd4a1db58013d4863b8992' move scheduled
I20240502 01:55:49.915306  8464 rebalancer_tool.cc:1084] tablet 89f7cf12d85442749e7d5b259ebca43f: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '60298ffea1dd4c25a38765c2ec9ec9d5' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240502 01:55:49.915511  8464 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:55:51.696786  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5606c3da105b4b86868c6c51ecdd39aa: could not find any suitable replica to move from server 218bc8c1f9ce46059898fc738179adf9 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:55:51.697072  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ef6f7a2e933049e5982deec12e9a670a: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 436822a397664c54b8d1b9eb3bc6cff4
W20240502 01:55:51.697207  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fcde569cfdf747c2aaecac7d265b4b1f: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 60298ffea1dd4c25a38765c2ec9ec9d5
W20240502 01:55:51.699545  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1ed4a88787d6401481245c552170a456: could not find any suitable replica to move from server 218bc8c1f9ce46059898fc738179adf9 to server 436822a397664c54b8d1b9eb3bc6cff4
W20240502 01:55:51.699829  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 63a163be603345428cc84c86b96467f0: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:55:51.700016  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fcde569cfdf747c2aaecac7d265b4b1f: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:55:51.700191  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ef7ebedfbbff4054b074bf7a7622b632: could not find any suitable replica to move from server 218bc8c1f9ce46059898fc738179adf9 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:55:51.700343  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f406d4a60a5e40c485796d37ff4d76bc: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 60298ffea1dd4c25a38765c2ec9ec9d5
W20240502 01:55:51.700457  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1ed4a88787d6401481245c552170a456: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:55:51.700575  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fcde569cfdf747c2aaecac7d265b4b1f: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 436822a397664c54b8d1b9eb3bc6cff4
W20240502 01:55:51.700690  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c2701645b8bc497aaf63e970157bdd80: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:55:51.700824  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0053d72412224a79be83b7e007b51934: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 436822a397664c54b8d1b9eb3bc6cff4
W20240502 01:55:51.700940  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5606c3da105b4b86868c6c51ecdd39aa: could not find any suitable replica to move from server 218bc8c1f9ce46059898fc738179adf9 to server 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:55:51.701062  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c02dbeef0b7241e78a12e4bd7ef79e47: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:55:51.701169  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ef6f7a2e933049e5982deec12e9a670a: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 60298ffea1dd4c25a38765c2ec9ec9d5
W20240502 01:55:51.701275  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 63a163be603345428cc84c86b96467f0: could not find any suitable replica to move from server 218bc8c1f9ce46059898fc738179adf9 to server 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:55:51.701386  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ef7ebedfbbff4054b074bf7a7622b632: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 60298ffea1dd4c25a38765c2ec9ec9d5
W20240502 01:55:51.701509  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f406d4a60a5e40c485796d37ff4d76bc: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:55:51.701637  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1ed4a88787d6401481245c552170a456: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 436822a397664c54b8d1b9eb3bc6cff4
I20240502 01:55:51.774039  8464 rebalancer_tool.cc:1074] tablet ac3325a06d8841718158e1f5f181a398: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '' move scheduled
I20240502 01:55:52.331655  8464 rebalancer_tool.cc:1074] tablet 73ed91df7ff04846991c95b30a86ea7e: '07cdb213c77c4ffda1f999b66b725399' -> '436822a397664c54b8d1b9eb3bc6cff4' move scheduled
I20240502 01:55:52.855840  8464 rebalancer_tool.cc:1074] tablet 74eadf4c939c4c6383732cd74c5ec9d6: '218bc8c1f9ce46059898fc738179adf9' -> '60298ffea1dd4c25a38765c2ec9ec9d5' move scheduled
I20240502 01:55:53.530508  8464 rebalancer_tool.cc:1074] tablet 5be45a68cff24a75a03b2fcd4b7851f9: '94ffd7aaf85b4e17881241b8396fa2f3' -> '9db17f84d37944098cbeb9e0d8456efb' move scheduled
I20240502 01:55:53.959184  8464 rebalancer_tool.cc:1074] tablet 2cefdfa8851147999c1425c606f36b1f: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '6d34e50fb5dd4a1db58013d4863b8992' move scheduled
I20240502 01:55:54.112434  8464 rebalancer_tool.cc:1074] tablet e4d4a1bd80f44b668a2d504e47fb744a: '94ffd7aaf85b4e17881241b8396fa2f3' -> '' move scheduled
I20240502 01:55:54.318883  8464 rebalancer_tool.cc:1084] tablet afa5561f8c0e4c0fa9bbbd5da61348c7: '07cdb213c77c4ffda1f999b66b725399' -> '436822a397664c54b8d1b9eb3bc6cff4' move ignored: Illegal state: tablet afa5561f8c0e4c0fa9bbbd5da61348c7: replica 436822a397664c54b8d1b9eb3bc6cff4 is already present
I20240502 01:55:54.319022  8464 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:55:56.646891  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c02dbeef0b7241e78a12e4bd7ef79e47: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:55:56.647063  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fcde569cfdf747c2aaecac7d265b4b1f: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:55:56.647253  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c2701645b8bc497aaf63e970157bdd80: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:55:56.647346  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ef6f7a2e933049e5982deec12e9a670a: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 60298ffea1dd4c25a38765c2ec9ec9d5
W20240502 01:55:56.647532  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5606c3da105b4b86868c6c51ecdd39aa: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:55:56.647681  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f406d4a60a5e40c485796d37ff4d76bc: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:55:56.647850  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c02dbeef0b7241e78a12e4bd7ef79e47: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:55:56.677433  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c2701645b8bc497aaf63e970157bdd80: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 60298ffea1dd4c25a38765c2ec9ec9d5
I20240502 01:55:56.806878  8464 rebalancer_tool.cc:1084] tablet 866f87ea9f7f48edb667b1b2eb3fa239: '94ffd7aaf85b4e17881241b8396fa2f3' -> '436822a397664c54b8d1b9eb3bc6cff4' move ignored: Illegal state: tablet 866f87ea9f7f48edb667b1b2eb3fa239: replica 436822a397664c54b8d1b9eb3bc6cff4 is already present
I20240502 01:55:56.807021  8464 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:55:58.353646  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5606c3da105b4b86868c6c51ecdd39aa: could not find any suitable replica to move from server 218bc8c1f9ce46059898fc738179adf9 to server 60298ffea1dd4c25a38765c2ec9ec9d5
W20240502 01:55:58.354082  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ef6f7a2e933049e5982deec12e9a670a: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 436822a397664c54b8d1b9eb3bc6cff4
W20240502 01:55:58.354192  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fcde569cfdf747c2aaecac7d265b4b1f: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:55:58.354285  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 63a163be603345428cc84c86b96467f0: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 60298ffea1dd4c25a38765c2ec9ec9d5
W20240502 01:55:58.354410  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1ed4a88787d6401481245c552170a456: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 436822a397664c54b8d1b9eb3bc6cff4
W20240502 01:55:58.354508  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f406d4a60a5e40c485796d37ff4d76bc: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 60298ffea1dd4c25a38765c2ec9ec9d5
W20240502 01:55:58.354660  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c2701645b8bc497aaf63e970157bdd80: could not find any suitable replica to move from server 218bc8c1f9ce46059898fc738179adf9 to server 436822a397664c54b8d1b9eb3bc6cff4
W20240502 01:55:58.354763  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c02dbeef0b7241e78a12e4bd7ef79e47: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 6d34e50fb5dd4a1db58013d4863b8992
I20240502 01:55:58.502305  8464 rebalancer_tool.cc:1074] tablet ca458907378e42d48adaa0ee647db4cd: '07cdb213c77c4ffda1f999b66b725399' -> '' move scheduled
I20240502 01:55:59.009737  8464 rebalancer_tool.cc:1084] tablet 4f4088f3523f4cd0afdf0de0f447c7af: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '9db17f84d37944098cbeb9e0d8456efb' move ignored: Illegal state: Request specified cas_config_opid_index of 4 but the committed config has opid_index of 5
I20240502 01:55:59.009958  8464 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:56:00.357450  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5606c3da105b4b86868c6c51ecdd39aa: could not find any suitable replica to move from server 218bc8c1f9ce46059898fc738179adf9 to server 60298ffea1dd4c25a38765c2ec9ec9d5
W20240502 01:56:00.358088  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f406d4a60a5e40c485796d37ff4d76bc: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:56:00.358247  8464 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 63a163be603345428cc84c86b96467f0: could not find any suitable replica to move from server 218bc8c1f9ce46059898fc738179adf9 to server 6d34e50fb5dd4a1db58013d4863b8992
I20240502 01:56:00.468246  8464 rebalancer_tool.cc:1074] tablet a398da35f18c4df394db82c5bf89aeb1: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '' move scheduled
I20240502 01:56:00.609146  8464 rebalancer_tool.cc:1074] tablet 3a7698f6bd4349ae832a22c853d76fea: '218bc8c1f9ce46059898fc738179adf9' -> '' move scheduled
I20240502 01:56:02.908571  8464 rebalancer_tool.cc:1136] tablet a398da35f18c4df394db82c5bf89aeb1: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '' move completed: OK
I20240502 01:56:05.664507  8464 rebalancer_tool.cc:1136] tablet e4d4a1bd80f44b668a2d504e47fb744a: '94ffd7aaf85b4e17881241b8396fa2f3' -> '' move completed: OK
I20240502 01:56:16.376601  8464 rebalancer_tool.cc:1136] tablet 74eadf4c939c4c6383732cd74c5ec9d6: '218bc8c1f9ce46059898fc738179adf9' -> '60298ffea1dd4c25a38765c2ec9ec9d5' move completed: OK
I20240502 01:56:18.761159  8464 rebalancer_tool.cc:1074] tablet 265a9645e1f141cb88c786b9aef8f90e: '94ffd7aaf85b4e17881241b8396fa2f3' -> '' move scheduled
I20240502 01:56:18.995920  8464 rebalancer_tool.cc:1074] tablet d7922fd756344d14a336ff0c54afdf73: '07cdb213c77c4ffda1f999b66b725399' -> '' move scheduled
I20240502 01:56:19.060423  8464 rebalancer_tool.cc:1136] tablet 265a9645e1f141cb88c786b9aef8f90e: '94ffd7aaf85b4e17881241b8396fa2f3' -> '' move completed: OK
I20240502 01:56:19.234541  8464 rebalancer_tool.cc:1136] tablet 3a7698f6bd4349ae832a22c853d76fea: '218bc8c1f9ce46059898fc738179adf9' -> '' move completed: OK
I20240502 01:56:19.307742  8464 rebalancer_tool.cc:1136] tablet ca458907378e42d48adaa0ee647db4cd: '07cdb213c77c4ffda1f999b66b725399' -> '' move completed: OK
I20240502 01:56:21.447381  8464 rebalancer_tool.cc:1136] tablet ac3325a06d8841718158e1f5f181a398: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '' move completed: OK
I20240502 01:56:21.645949  8464 rebalancer_tool.cc:1136] tablet 9a5f486852554e2fa2d07f7b9d686ced: '218bc8c1f9ce46059898fc738179adf9' -> '6d34e50fb5dd4a1db58013d4863b8992' move completed: OK
I20240502 01:56:21.806586  8464 rebalancer_tool.cc:1136] tablet 73ed91df7ff04846991c95b30a86ea7e: '07cdb213c77c4ffda1f999b66b725399' -> '436822a397664c54b8d1b9eb3bc6cff4' move completed: OK
I20240502 01:56:21.851415  8464 rebalancer_tool.cc:1136] tablet 5be45a68cff24a75a03b2fcd4b7851f9: '94ffd7aaf85b4e17881241b8396fa2f3' -> '9db17f84d37944098cbeb9e0d8456efb' move completed: OK
I20240502 01:56:21.942845  8464 rebalancer_tool.cc:1136] tablet 2cefdfa8851147999c1425c606f36b1f: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '6d34e50fb5dd4a1db58013d4863b8992' move completed: OK
I20240502 01:56:22.043303  8464 rebalancer_tool.cc:1074] tablet 02b106ec57b94e44a29947a80a030a23: '218bc8c1f9ce46059898fc738179adf9' -> '' move scheduled
I20240502 01:56:22.348438  8464 rebalancer_tool.cc:1074] tablet 4eb91f5df851459b8ecf09cbe34fefa5: '94ffd7aaf85b4e17881241b8396fa2f3' -> '' move scheduled
I20240502 01:56:22.611953  8464 rebalancer_tool.cc:1074] tablet f309b40931dd444091869119eab33bbc: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '' move scheduled
I20240502 01:56:22.762284  8464 rebalancer_tool.cc:1074] tablet 0cecd1b14e474dee825fae040ce1b31e: '07cdb213c77c4ffda1f999b66b725399' -> '' move scheduled
I20240502 01:56:26.589659  8464 rebalancer_tool.cc:1136] tablet 4eb91f5df851459b8ecf09cbe34fefa5: '94ffd7aaf85b4e17881241b8396fa2f3' -> '' move completed: OK
I20240502 01:56:26.705051  8464 rebalancer_tool.cc:1136] tablet 02b106ec57b94e44a29947a80a030a23: '218bc8c1f9ce46059898fc738179adf9' -> '' move completed: OK
I20240502 01:56:26.790154  8464 rebalancer_tool.cc:1136] tablet d7922fd756344d14a336ff0c54afdf73: '07cdb213c77c4ffda1f999b66b725399' -> '' move completed: OK
I20240502 01:56:26.901233  8464 rebalancer_tool.cc:1074] tablet 7f2519b0c6404799ad84d120b45a7ea4: '94ffd7aaf85b4e17881241b8396fa2f3' -> '' move scheduled
I20240502 01:56:26.982676  8464 rebalancer_tool.cc:1074] tablet c58d6b2401c047cd9af56a66c2e00105: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '' move scheduled
I20240502 01:56:27.159976  8464 rebalancer_tool.cc:1074] tablet 99e74a49b42745888058db67049dc682: '07cdb213c77c4ffda1f999b66b725399' -> '' move scheduled
I20240502 01:56:27.259637  8464 rebalancer_tool.cc:1136] tablet 99e74a49b42745888058db67049dc682: '07cdb213c77c4ffda1f999b66b725399' -> '' move completed: OK
I20240502 01:56:27.300731  8464 rebalancer_tool.cc:1136] tablet c58d6b2401c047cd9af56a66c2e00105: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '' move completed: OK
I20240502 01:56:33.477468  8464 rebalancer_tool.cc:1074] tablet 4f4088f3523f4cd0afdf0de0f447c7af: '94ffd7aaf85b4e17881241b8396fa2f3' -> '' move scheduled
I20240502 01:56:33.553122  8464 rebalancer_tool.cc:1074] tablet afa5561f8c0e4c0fa9bbbd5da61348c7: '07cdb213c77c4ffda1f999b66b725399' -> '' move scheduled
I20240502 01:56:33.667668  8464 rebalancer_tool.cc:1136] tablet afa5561f8c0e4c0fa9bbbd5da61348c7: '07cdb213c77c4ffda1f999b66b725399' -> '' move completed: OK
I20240502 01:56:33.708370  8464 rebalancer_tool.cc:1136] tablet 4f4088f3523f4cd0afdf0de0f447c7af: '94ffd7aaf85b4e17881241b8396fa2f3' -> '' move completed: OK
I20240502 01:56:37.932220  8464 rebalancer_tool.cc:1074] tablet 89f7cf12d85442749e7d5b259ebca43f: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '' move scheduled
I20240502 01:56:37.963286  8464 rebalancer_tool.cc:1074] tablet 4486b413e5c14ec6953e4895ee5797cf: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '' move scheduled
I20240502 01:56:39.229903  8464 rebalancer_tool.cc:1136] tablet 89f7cf12d85442749e7d5b259ebca43f: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '' move completed: OK
W20240502 01:56:40.433081 17916 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 01:56:40.060853 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.136:44359 (local address 127.0.0.1:38486)
0502 01:56:40.061365 (+   512us) negotiation.cc:107] Waiting for socket to connect
0502 01:56:40.061384 (+    19us) client_negotiation.cc:174] Beginning negotiation
0502 01:56:40.061529 (+   145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 01:56:40.431714 (+370185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":390}
I20240502 01:56:43.450928  8464 rebalancer_tool.cc:1136] tablet 4486b413e5c14ec6953e4895ee5797cf: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '' move completed: OK
W20240502 01:58:40.588204  3777 meta_cache.cc:1261] Time spent looking up entry by key: real 0.051s	user 0.000s	sys 0.000s
I20240502 02:01:28.865036  8464 rebalancer_tool.cc:1136] tablet 0cecd1b14e474dee825fae040ce1b31e: '07cdb213c77c4ffda1f999b66b725399' -> '' move completed: OK
I20240502 02:01:28.974673  8464 rebalancer_tool.cc:1136] tablet f309b40931dd444091869119eab33bbc: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '' move completed: OK
I20240502 02:01:29.039043  8464 rebalancer_tool.cc:1136] tablet 7f2519b0c6404799ad84d120b45a7ea4: '94ffd7aaf85b4e17881241b8396fa2f3' -> '' move completed: OK
Illegal state: tablet server 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): unacceptable health status UNAVAILABLE

W20240502 02:01:30.168516  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:01:30.202369  5999 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 02:01:30.217248  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:30.225643  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:30.237509  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:30.253813  6446 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:30.255249  6180 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:30.257247  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:30.385236  6335 consensus_queue.cc:237] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 1.17, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 02:01:30.387324  8268 raft_consensus.cc:1261] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 17. Preceding OpId from leader: term: 1 index: 18. (index mismatch)
I20240502 02:01:30.387930 29767 consensus_queue.cc:1035] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20240502 02:01:30.390591 29218 raft_consensus.cc:2932] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Committing config change with OpId 1.18: config changed from index 17 to 18, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 02:01:30.391407  8268 raft_consensus.cc:2932] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Committing config change with OpId 1.18: config changed from index 17 to 18, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 02:01:30.397156  5917 catalog_manager.cc:5484] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 reported cstate change: config changed from index 17 to 18, VOTER 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129) evicted. New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } }
I20240502 02:01:30.404565  6048 tablet_service.cc:1514] Processing DeleteTablet for tablet afa5561f8c0e4c0fa9bbbd5da61348c7 with delete_type TABLET_DATA_TOMBSTONED (TS 218bc8c1f9ce46059898fc738179adf9 not found in new config with opid_index 18) from {username='slave'} at 127.0.0.1:58306
I20240502 02:01:30.405398 29790 tablet_replica.cc:330] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9: stopping tablet replica
I20240502 02:01:30.405630 29790 raft_consensus.cc:2229] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 02:01:30.405802 29790 raft_consensus.cc:2258] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 02:01:30.405704  6335 consensus_queue.cc:237] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 1.18, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 02:01:30.406589 29790 ts_tablet_manager.cc:1875] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 02:01:30.407696  8268 raft_consensus.cc:1261] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Refusing update from remote peer 94ffd7aaf85b4e17881241b8396fa2f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 18. Preceding OpId from leader: term: 1 index: 19. (index mismatch)
W20240502 02:01:30.408082  6271 consensus_peers.cc:485] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197): Couldn't send request to peer a2c90ee440b44b708bccc2e2d5385cc0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: afa5561f8c0e4c0fa9bbbd5da61348c7. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:01:30.408258 29218 consensus_queue.cc:1035] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20240502 02:01:30.410478 29767 raft_consensus.cc:2932] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3 [term 1 LEADER]: Committing config change with OpId 1.19: config changed from index 18 to 19, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) added. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } }
I20240502 02:01:30.414115 29790 ts_tablet_manager.cc:1888] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.17
I20240502 02:01:30.414242 29790 log.cc:1196] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 218bc8c1f9ce46059898fc738179adf9: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal/wals/afa5561f8c0e4c0fa9bbbd5da61348c7
I20240502 02:01:30.414536  5907 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet afa5561f8c0e4c0fa9bbbd5da61348c7 with cas_config_opid_index 18: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 02:01:30.414860  5907 catalog_manager.cc:4830] TS 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227): tablet afa5561f8c0e4c0fa9bbbd5da61348c7 (table rebalance_test_table_2 [id=ef7ebedfbbff4054b074bf7a7622b632]) successfully deleted
I20240502 02:01:30.415038  8268 raft_consensus.cc:2932] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Committing config change with OpId 1.19: config changed from index 18 to 19, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) added. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } }
I20240502 02:01:30.420539  5917 catalog_manager.cc:5484] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 reported cstate change: config changed from index 18 to 19, VOTER a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130) added. New cstate: current_term: 1 leader_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" committed_config { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } }
W20240502 02:01:30.444747  6135 consensus_peers.cc:485] T 086509f1ee0a4a81b2f8194b50dbe159 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 02:01:30.508555  5907 catalog_manager.cc:4573] TS 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet afa5561f8c0e4c0fa9bbbd5da61348c7: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:01:30.535728  6401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240502 02:01:30.536604  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:01:30.593825  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:01:30.607606  6402 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
I20240502 02:01:30.680931 29218 raft_consensus.cc:482] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6d34e50fb5dd4a1db58013d4863b8992)
I20240502 02:01:30.681145 29218 raft_consensus.cc:504] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "9db17f84d37944098cbeb9e0d8456efb" member_type: VOTER last_known_addr { host: "127.25.109.134" port: 42517 } }
I20240502 02:01:30.681959 29218 leader_election.cc:288] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359), 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517)
W20240502 02:01:30.683790  6269 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:01:30.685565  6270 leader_election.cc:334] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
I20240502 02:01:30.685674  6270 leader_election.cc:302] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94ffd7aaf85b4e17881241b8396fa2f3; no voters: 6d34e50fb5dd4a1db58013d4863b8992, 9db17f84d37944098cbeb9e0d8456efb
I20240502 02:01:30.685923 29218 raft_consensus.cc:2726] T d8a6db6ae11e4cb8881aee95bdb0e59d P 94ffd7aaf85b4e17881241b8396fa2f3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 02:01:30.779449  6136 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
I20240502 02:01:30.787468  6446 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:30.804467  8248 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:30.824813  6180 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:30.862771  6314 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 02:01:30.868167  7981 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:30.869910  6048 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:30.918201 29843 ts_tablet_manager.cc:902] T afa5561f8c0e4c0fa9bbbd5da61348c7 P a2c90ee440b44b708bccc2e2d5385cc0: Initiating tablet copy from peer 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245)
I20240502 02:01:30.947440 29843 tablet_copy_client.cc:308] T afa5561f8c0e4c0fa9bbbd5da61348c7 P a2c90ee440b44b708bccc2e2d5385cc0: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.131:44245
I20240502 02:01:30.948264  6357 tablet_copy_service.cc:140] P 94ffd7aaf85b4e17881241b8396fa2f3: Received BeginTabletCopySession request for tablet afa5561f8c0e4c0fa9bbbd5da61348c7 from peer a2c90ee440b44b708bccc2e2d5385cc0 ({username='slave'} at 127.25.109.130:37051)
I20240502 02:01:30.948417  6357 tablet_copy_service.cc:161] P 94ffd7aaf85b4e17881241b8396fa2f3: Beginning new tablet copy session on tablet afa5561f8c0e4c0fa9bbbd5da61348c7 from peer a2c90ee440b44b708bccc2e2d5385cc0 at {username='slave'} at 127.25.109.130:37051: session id = a2c90ee440b44b708bccc2e2d5385cc0-afa5561f8c0e4c0fa9bbbd5da61348c7
I20240502 02:01:30.952299  6357 tablet_copy_source_session.cc:205] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 94ffd7aaf85b4e17881241b8396fa2f3: Tablet Copy: opened 5 blocks and 1 log segments
W20240502 02:01:30.953402  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:01:30.954113 29843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afa5561f8c0e4c0fa9bbbd5da61348c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:30.961829 29843 tablet_copy_client.cc:783] T afa5561f8c0e4c0fa9bbbd5da61348c7 P a2c90ee440b44b708bccc2e2d5385cc0: tablet copy: Starting download of 5 data blocks...
I20240502 02:01:30.969421 29843 tablet_copy_client.cc:647] T afa5561f8c0e4c0fa9bbbd5da61348c7 P a2c90ee440b44b708bccc2e2d5385cc0: tablet copy: Starting download of 1 WAL segments...
I20240502 02:01:30.971367 29843 tablet_copy_client.cc:523] T afa5561f8c0e4c0fa9bbbd5da61348c7 P a2c90ee440b44b708bccc2e2d5385cc0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:01:30.975800 29843 tablet_bootstrap.cc:492] T afa5561f8c0e4c0fa9bbbd5da61348c7 P a2c90ee440b44b708bccc2e2d5385cc0: Bootstrap starting.
I20240502 02:01:31.002007 29843 tablet_bootstrap.cc:492] T afa5561f8c0e4c0fa9bbbd5da61348c7 P a2c90ee440b44b708bccc2e2d5385cc0: Bootstrap replayed 1/1 log segments. Stats: ops{read=19 overwritten=0 applied=19 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:01:31.002374 29843 tablet_bootstrap.cc:492] T afa5561f8c0e4c0fa9bbbd5da61348c7 P a2c90ee440b44b708bccc2e2d5385cc0: Bootstrap complete.
I20240502 02:01:31.002622 29843 ts_tablet_manager.cc:1367] T afa5561f8c0e4c0fa9bbbd5da61348c7 P a2c90ee440b44b708bccc2e2d5385cc0: Time spent bootstrapping tablet: real 0.027s	user 0.025s	sys 0.003s
I20240502 02:01:31.003291 29843 raft_consensus.cc:348] T afa5561f8c0e4c0fa9bbbd5da61348c7 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 02:01:31.003456 29843 raft_consensus.cc:729] T afa5561f8c0e4c0fa9bbbd5da61348c7 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c90ee440b44b708bccc2e2d5385cc0, State: Initialized, Role: FOLLOWER
I20240502 02:01:31.003706 29843 consensus_queue.cc:260] T afa5561f8c0e4c0fa9bbbd5da61348c7 P a2c90ee440b44b708bccc2e2d5385cc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 19, Last appended: 1.19, Last appended by leader: 19, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 02:01:31.004240 29843 ts_tablet_manager.cc:1398] T afa5561f8c0e4c0fa9bbbd5da61348c7 P a2c90ee440b44b708bccc2e2d5385cc0: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240502 02:01:31.005048  6357 tablet_copy_service.cc:342] P 94ffd7aaf85b4e17881241b8396fa2f3: Request end of tablet copy session a2c90ee440b44b708bccc2e2d5385cc0-afa5561f8c0e4c0fa9bbbd5da61348c7 received from {username='slave'} at 127.25.109.130:37051
I20240502 02:01:31.005151  6357 tablet_copy_service.cc:433] P 94ffd7aaf85b4e17881241b8396fa2f3: ending tablet copy session a2c90ee440b44b708bccc2e2d5385cc0-afa5561f8c0e4c0fa9bbbd5da61348c7 on tablet afa5561f8c0e4c0fa9bbbd5da61348c7 with peer a2c90ee440b44b708bccc2e2d5385cc0
W20240502 02:01:31.106128  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:01:31.118321  6135 consensus_peers.cc:485] T ca458907378e42d48adaa0ee647db4cd P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 02:01:31.126893  6251 maintenance_manager.cc:419] P a2c90ee440b44b708bccc2e2d5385cc0: Scheduling UndoDeltaBlockGCOp(afa5561f8c0e4c0fa9bbbd5da61348c7): 4581 bytes on disk
I20240502 02:01:31.127441  6138 maintenance_manager.cc:643] P a2c90ee440b44b708bccc2e2d5385cc0: UndoDeltaBlockGCOp(afa5561f8c0e4c0fa9bbbd5da61348c7) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
W20240502 02:01:31.154379  6000 consensus_peers.cc:485] T c58d6b2401c047cd9af56a66c2e00105 P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240502 02:01:31.238637  6000 consensus_peers.cc:485] T 4f4088f3523f4cd0afdf0de0f447c7af P 218bc8c1f9ce46059898fc738179adf9 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Couldn't send request to peer 9db17f84d37944098cbeb9e0d8456efb. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240502 02:01:31.293905  6269 consensus_peers.cc:485] T 3a7698f6bd4349ae832a22c853d76fea P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 02:01:31.316493  6200 raft_consensus.cc:1203] T afa5561f8c0e4c0fa9bbbd5da61348c7 P a2c90ee440b44b708bccc2e2d5385cc0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.18->[1.19-1.19]   Dedup: 1.19->[]
W20240502 02:01:31.417570  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:01:31.422824  8458 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240502 01:55:43.556095  8461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:43.556567  8461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:43.568176  8461 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:55:47.064046  8461 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 01:55:48.742746  8461 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5606c3da105b4b86868c6c51ecdd39aa: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:55:48.815819  8461 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ef7ebedfbbff4054b074bf7a7622b632: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:55:48.820210  8461 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fcde569cfdf747c2aaecac7d265b4b1f: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:55:48.820789  8461 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c2701645b8bc497aaf63e970157bdd80: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:55:48.821398  8461 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0053d72412224a79be83b7e007b51934: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:55:48.821841  8461 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c02dbeef0b7241e78a12e4bd7ef79e47: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 60298ffea1dd4c25a38765c2ec9ec9d5
W20240502 01:55:48.822041  8461 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f406d4a60a5e40c485796d37ff4d76bc: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:55:48.822175  8461 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ef6f7a2e933049e5982deec12e9a670a: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:55:48.822288  8461 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1ed4a88787d6401481245c552170a456: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 436822a397664c54b8d1b9eb3bc6cff4
W20240502 01:55:48.822402  8461 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 63a163be603345428cc84c86b96467f0: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:55:48.822510  8461 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fcde569cfdf747c2aaecac7d265b4b1f: could not find any suitable replica to move from server 218bc8c1f9ce46059898fc738179adf9 to server 436822a397664c54b8d1b9eb3bc6cff4
W20240502 01:55:48.822671  8461 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c2701645b8bc497aaf63e970157bdd80: could not find any suitable replica to move from server 218bc8c1f9ce46059898fc738179adf9 to server 6d34e50fb5dd4a1db58013d4863b8992
W20240502 01:55:48.822793  8461 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c02dbeef0b7241e78a12e4bd7ef79e47: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 60298ffea1dd4c25a38765c2ec9ec9d5
W20240502 01:55:48.822908  8461 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ef6f7a2e933049e5982deec12e9a670a: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 436822a397664c54b8d1b9eb3bc6cff4
W20240502 01:55:48.823009  8461 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0053d72412224a79be83b7e007b51934: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:55:48.823139  8461 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5606c3da105b4b86868c6c51ecdd39aa: could not find any suitable replica to move from server 218bc8c1f9ce46059898fc738179adf9 to server 6d34e50fb5dd4a1db58013d4863b8992
I20240502 01:55:49.310631  8461 rebalancer_tool.cc:1074] tablet 0cecd1b14e474dee825fae040ce1b31e: '218bc8c1f9ce46059898fc738179adf9' -> '60298ffea1dd4c25a38765c2ec9ec9d5' move scheduled
I20240502 01:55:49.708830  8461 rebalancer_tool.cc:1074] tablet 89f7cf12d85442749e7d5b259ebca43f: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '436822a397664c54b8d1b9eb3bc6cff4' move scheduled
I20240502 01:55:50.434648  8461 rebalancer_tool.cc:1074] tablet 899963a8260a42c4b73fb80eda2ecaa9: '07cdb213c77c4ffda1f999b66b725399' -> '9db17f84d37944098cbeb9e0d8456efb' move scheduled
I20240502 01:55:50.749549  8461 rebalancer_tool.cc:1074] tablet d7922fd756344d14a336ff0c54afdf73: '94ffd7aaf85b4e17881241b8396fa2f3' -> '6d34e50fb5dd4a1db58013d4863b8992' move scheduled
I20240502 01:55:51.322816  8461 rebalancer_tool.cc:1074] tablet 3a7698f6bd4349ae832a22c853d76fea: '218bc8c1f9ce46059898fc738179adf9' -> '60298ffea1dd4c25a38765c2ec9ec9d5' move scheduled
I20240502 01:55:51.748368  8461 rebalancer_tool.cc:1074] tablet a398da35f18c4df394db82c5bf89aeb1: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '436822a397664c54b8d1b9eb3bc6cff4' move scheduled
I20240502 01:55:52.611153  8461 rebalancer_tool.cc:1074] tablet ca458907378e42d48adaa0ee647db4cd: '07cdb213c77c4ffda1f999b66b725399' -> '9db17f84d37944098cbeb9e0d8456efb' move scheduled
I20240502 01:55:53.426025  8461 rebalancer_tool.cc:1074] tablet 4eb91f5df851459b8ecf09cbe34fefa5: '94ffd7aaf85b4e17881241b8396fa2f3' -> '6d34e50fb5dd4a1db58013d4863b8992' move scheduled
I20240502 01:55:53.839852  8461 rebalancer_tool.cc:1074] tablet 02b106ec57b94e44a29947a80a030a23: '218bc8c1f9ce46059898fc738179adf9' -> '60298ffea1dd4c25a38765c2ec9ec9d5' move scheduled
I20240502 01:55:54.369877  8461 rebalancer_tool.cc:1074] tablet 2cefdfa8851147999c1425c606f36b1f: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '436822a397664c54b8d1b9eb3bc6cff4' move scheduled
I20240502 01:55:54.812606  8461 rebalancer_tool.cc:1074] tablet 086509f1ee0a4a81b2f8194b50dbe159: '07cdb213c77c4ffda1f999b66b725399' -> '6d34e50fb5dd4a1db58013d4863b8992' move scheduled
I20240502 01:55:55.194439  8461 rebalancer_tool.cc:1074] tablet 6984d638991442e3adaed1b24bee95e9: '94ffd7aaf85b4e17881241b8396fa2f3' -> '9db17f84d37944098cbeb9e0d8456efb' move scheduled
I20240502 01:55:55.302839  8461 rebalancer_tool.cc:1084] tablet afa5561f8c0e4c0fa9bbbd5da61348c7: '218bc8c1f9ce46059898fc738179adf9' -> '436822a397664c54b8d1b9eb3bc6cff4' move ignored: Illegal state: tablet afa5561f8c0e4c0fa9bbbd5da61348c7: replica 436822a397664c54b8d1b9eb3bc6cff4 is already present
I20240502 01:55:55.303048  8461 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:55:56.744290  8461 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c02dbeef0b7241e78a12e4bd7ef79e47: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 60298ffea1dd4c25a38765c2ec9ec9d5
W20240502 01:55:56.744511  8461 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 63a163be603345428cc84c86b96467f0: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 436822a397664c54b8d1b9eb3bc6cff4
W20240502 01:55:56.744840  8461 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0053d72412224a79be83b7e007b51934: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 60298ffea1dd4c25a38765c2ec9ec9d5
W20240502 01:55:56.745005  8461 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1ed4a88787d6401481245c552170a456: could not find any suitable replica to move from server 218bc8c1f9ce46059898fc738179adf9 to server 60298ffea1dd4c25a38765c2ec9ec9d5
W20240502 01:55:56.745141  8461 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5606c3da105b4b86868c6c51ecdd39aa: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 436822a397664c54b8d1b9eb3bc6cff4
W20240502 01:55:56.747486  8461 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c02dbeef0b7241e78a12e4bd7ef79e47: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 9db17f84d37944098cbeb9e0d8456efb
W20240502 01:55:56.747637  8461 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fcde569cfdf747c2aaecac7d265b4b1f: could not find any suitable replica to move from server 218bc8c1f9ce46059898fc738179adf9 to server 436822a397664c54b8d1b9eb3bc6cff4
W20240502 01:55:56.747797  8461 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c2701645b8bc497aaf63e970157bdd80: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 6d34e50fb5dd4a1db58013d4863b8992
I20240502 01:55:56.870174  8461 rebalancer_tool.cc:1074] tablet 73ed91df7ff04846991c95b30a86ea7e: '218bc8c1f9ce46059898fc738179adf9' -> '' move scheduled
I20240502 01:55:56.997267  8461 rebalancer_tool.cc:1074] tablet f309b40931dd444091869119eab33bbc: '94ffd7aaf85b4e17881241b8396fa2f3' -> '' move scheduled
I20240502 01:55:57.116954  8461 rebalancer_tool.cc:1074] tablet 99e74a49b42745888058db67049dc682: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '' move scheduled
I20240502 01:55:57.518750  8461 rebalancer_tool.cc:1074] tablet 55de74ae85e541a19008fd2fc2da3ffd: '218bc8c1f9ce46059898fc738179adf9' -> '436822a397664c54b8d1b9eb3bc6cff4' move scheduled
I20240502 01:55:58.003186  8461 rebalancer_tool.cc:1074] tablet 4f4088f3523f4cd0afdf0de0f447c7af: '94ffd7aaf85b4e17881241b8396fa2f3' -> '6d34e50fb5dd4a1db58013d4863b8992' move scheduled
I20240502 01:55:58.302207  8461 rebalancer_tool.cc:1084] tablet 265a9645e1f141cb88c786b9aef8f90e: '07cdb213c77c4ffda1f999b66b725399' -> '9db17f84d37944098cbeb9e0d8456efb' move ignored: Illegal state: Request specified cas_config_opid_index of 4 but the committed config has opid_index of 5
I20240502 01:55:58.302397  8461 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 01:56:00.045046  8461 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c02dbeef0b7241e78a12e4bd7ef79e47: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 60298ffea1dd4c25a38765c2ec9ec9d5
W20240502 01:56:00.046424  8461 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0053d72412224a79be83b7e007b51934: could not find any suitable replica to move from server a2c90ee440b44b708bccc2e2d5385cc0 to server 60298ffea1dd4c25a38765c2ec9ec9d5
W20240502 01:56:00.046648  8461 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5606c3da105b4b86868c6c51ecdd39aa: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 436822a397664c54b8d1b9eb3bc6cff4
W20240502 01:56:00.046787  8461 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 63a163be603345428cc84c86b96467f0: could not find any suitable replica to move from server 94ffd7aaf85b4e17881241b8396fa2f3 to server 436822a397664c54b8d1b9eb3bc6cff4
W20240502 01:56:00.046944  8461 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f406d4a60a5e40c485796d37ff4d76bc: could not find any suitable replica to move from server 07cdb213c77c4ffda1f999b66b725399 to server 60298ffea1dd4c25a38765c2ec9ec9d5
I20240502 01:56:00.255995  8461 rebalancer_tool.cc:1074] tablet d8a6db6ae11e4cb8881aee95bdb0e59d: '07cdb213c77c4ffda1f999b66b725399' -> '' move scheduled
I20240502 01:56:00.378515  8461 rebalancer_tool.cc:1074] tablet c58d6b2401c047cd9af56a66c2e00105: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '' move scheduled
I20240502 01:56:07.222177  8461 rebalancer_tool.cc:1136] tablet a398da35f18c4df394db82c5bf89aeb1: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '436822a397664c54b8d1b9eb3bc6cff4' move completed: OK
I20240502 01:56:07.295285  8461 rebalancer_tool.cc:1136] tablet d7922fd756344d14a336ff0c54afdf73: '94ffd7aaf85b4e17881241b8396fa2f3' -> '6d34e50fb5dd4a1db58013d4863b8992' move completed: OK
I20240502 01:56:07.424993  8461 rebalancer_tool.cc:1136] tablet 899963a8260a42c4b73fb80eda2ecaa9: '07cdb213c77c4ffda1f999b66b725399' -> '9db17f84d37944098cbeb9e0d8456efb' move completed: OK
I20240502 01:56:12.504797  8461 rebalancer_tool.cc:1136] tablet ca458907378e42d48adaa0ee647db4cd: '07cdb213c77c4ffda1f999b66b725399' -> '9db17f84d37944098cbeb9e0d8456efb' move completed: OK
I20240502 01:56:21.330591  8461 rebalancer_tool.cc:1136] tablet 3a7698f6bd4349ae832a22c853d76fea: '218bc8c1f9ce46059898fc738179adf9' -> '60298ffea1dd4c25a38765c2ec9ec9d5' move completed: OK
I20240502 01:56:21.544493  8461 rebalancer_tool.cc:1136] tablet 6984d638991442e3adaed1b24bee95e9: '94ffd7aaf85b4e17881241b8396fa2f3' -> '9db17f84d37944098cbeb9e0d8456efb' move completed: OK
I20240502 01:56:23.162740  8461 rebalancer_tool.cc:1136] tablet 73ed91df7ff04846991c95b30a86ea7e: '218bc8c1f9ce46059898fc738179adf9' -> '' move completed: OK
I20240502 01:56:23.208576  8461 rebalancer_tool.cc:1136] tablet 2cefdfa8851147999c1425c606f36b1f: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '436822a397664c54b8d1b9eb3bc6cff4' move completed: OK
I20240502 01:56:23.251024  8461 rebalancer_tool.cc:1136] tablet 99e74a49b42745888058db67049dc682: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '' move completed: OK
I20240502 01:56:23.305219  8461 rebalancer_tool.cc:1074] tablet 265a9645e1f141cb88c786b9aef8f90e: '07cdb213c77c4ffda1f999b66b725399' -> '' move scheduled
I20240502 01:56:23.353011  8461 rebalancer_tool.cc:1074] tablet 0039dd405cfc4dcea5e7353169e3cd52: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '' move scheduled
I20240502 01:56:23.485446  8461 rebalancer_tool.cc:1136] tablet 265a9645e1f141cb88c786b9aef8f90e: '07cdb213c77c4ffda1f999b66b725399' -> '' move completed: OK
I20240502 01:56:23.573630  8461 rebalancer_tool.cc:1136] tablet c58d6b2401c047cd9af56a66c2e00105: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '' move completed: OK
I20240502 01:56:23.684319  8461 rebalancer_tool.cc:1136] tablet d8a6db6ae11e4cb8881aee95bdb0e59d: '07cdb213c77c4ffda1f999b66b725399' -> '' move completed: OK
I20240502 01:56:23.754727  8461 rebalancer_tool.cc:1136] tablet 4f4088f3523f4cd0afdf0de0f447c7af: '94ffd7aaf85b4e17881241b8396fa2f3' -> '6d34e50fb5dd4a1db58013d4863b8992' move completed: OK
I20240502 01:56:23.935907  8461 rebalancer_tool.cc:1136] tablet 086509f1ee0a4a81b2f8194b50dbe159: '07cdb213c77c4ffda1f999b66b725399' -> '6d34e50fb5dd4a1db58013d4863b8992' move completed: OK
I20240502 01:56:24.057606  8461 rebalancer_tool.cc:1136] tablet 0cecd1b14e474dee825fae040ce1b31e: '218bc8c1f9ce46059898fc738179adf9' -> '60298ffea1dd4c25a38765c2ec9ec9d5' move completed: OK
I20240502 01:56:24.170980  8461 rebalancer_tool.cc:1136] tablet 4eb91f5df851459b8ecf09cbe34fefa5: '94ffd7aaf85b4e17881241b8396fa2f3' -> '6d34e50fb5dd4a1db58013d4863b8992' move completed: OK
I20240502 01:56:26.343713  8461 rebalancer_tool.cc:1136] tablet 55de74ae85e541a19008fd2fc2da3ffd: '218bc8c1f9ce46059898fc738179adf9' -> '436822a397664c54b8d1b9eb3bc6cff4' move completed: OK
I20240502 01:56:26.410938  8461 rebalancer_tool.cc:1136] tablet 89f7cf12d85442749e7d5b259ebca43f: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '436822a397664c54b8d1b9eb3bc6cff4' move completed: OK
I20240502 01:56:26.602308  8461 rebalancer_tool.cc:1136] tablet 02b106ec57b94e44a29947a80a030a23: '218bc8c1f9ce46059898fc738179adf9' -> '60298ffea1dd4c25a38765c2ec9ec9d5' move completed: OK
I20240502 01:56:26.646643  8461 rebalancer_tool.cc:1136] tablet f309b40931dd444091869119eab33bbc: '94ffd7aaf85b4e17881241b8396fa2f3' -> '' move completed: OK
I20240502 01:56:26.715922  8461 rebalancer_tool.cc:1074] tablet 9a5f486852554e2fa2d07f7b9d686ced: '218bc8c1f9ce46059898fc738179adf9' -> '' move scheduled
I20240502 01:56:26.784283  8461 rebalancer_tool.cc:1074] tablet 5be45a68cff24a75a03b2fcd4b7851f9: '94ffd7aaf85b4e17881241b8396fa2f3' -> '' move scheduled
I20240502 01:56:26.896363  8461 rebalancer_tool.cc:1074] tablet afa5561f8c0e4c0fa9bbbd5da61348c7: '218bc8c1f9ce46059898fc738179adf9' -> '' move scheduled
I20240502 01:56:26.956553  8461 rebalancer_tool.cc:1074] tablet e4d4a1bd80f44b668a2d504e47fb744a: '94ffd7aaf85b4e17881241b8396fa2f3' -> '' move scheduled
I20240502 01:56:27.178119  8461 rebalancer_tool.cc:1136] tablet e4d4a1bd80f44b668a2d504e47fb744a: '94ffd7aaf85b4e17881241b8396fa2f3' -> '' move completed: OK
I20240502 01:56:28.860059  8461 rebalancer_tool.cc:1136] tablet 5be45a68cff24a75a03b2fcd4b7851f9: '94ffd7aaf85b4e17881241b8396fa2f3' -> '' move completed: OK
I20240502 01:56:29.027644  8461 rebalancer_tool.cc:1136] tablet 9a5f486852554e2fa2d07f7b9d686ced: '218bc8c1f9ce46059898fc738179adf9' -> '' move completed: OK
I20240502 01:56:29.153789  8461 rebalancer_tool.cc:1136] tablet 0039dd405cfc4dcea5e7353169e3cd52: 'a2c90ee440b44b708bccc2e2d5385cc0' -> '' move completed: OK
W20240502 01:56:40.434720 17954 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 01:56:40.294039 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.109.136:44359 (local address 127.0.0.1:38494)
0502 01:56:40.295233 (+  1194us) negotiation.cc:107] Waiting for socket to connect
0502 01:56:40.295253 (+    20us) client_negotiation.cc:174] Beginning negotiation
0502 01:56:40.295380 (+   127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 01:56:40.431318 (+135938us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1075}
W20240502 01:57:19.566597 24278 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.25.109.129:36227, user_credentials={real_user=slave}} blocked reactor thread for 50415.6us
W20240502 02:00:45.433588 23063 meta_cache.cc:1261] Time spent looking up entry by key: real 0.132s	user 0.000s	sys 0.000s
W20240502 02:00:45.434126 23034 meta_cache.cc:1097] Time spent processing table locations response: real 0.132s	user 0.006s	sys 0.010s
W20240502 02:00:45.434154 23059 meta_cache.cc:1261] Time spent looking up entry by key: real 0.133s	user 0.000s	sys 0.000s
W20240502 02:00:45.434410 23034 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.25.109.190:37927, user_credentials={real_user=slave}} blocked reactor thread for 132641us
W20240502 02:00:45.437150 23057 meta_cache.cc:1261] Time spent looking up entry by key: real 0.135s	user 0.004s	sys 0.000s
I20240502 02:01:30.609098  8461 rebalancer_tool.cc:1136] tablet afa5561f8c0e4c0fa9bbbd5da61348c7: '218bc8c1f9ce46059898fc738179adf9' -> '' move completed: OK
Illegal state: tablet server 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): unacceptable health status UNAVAILABLE

E20240502 02:01:31.430400 26038 external_mini_cluster.cc:943] Process with UUID 9db17f84d37944098cbeb9e0d8456efb has crashed
E20240502 02:01:31.430532 26038 external_mini_cluster.cc:943] Process with UUID 6d34e50fb5dd4a1db58013d4863b8992 has crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/mini-cluster/external_mini_cluster.cc:947: Failure
Expected equality of these values:
  0
  num_crashes
    Which is: 2
At least one process crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/rebalancer_tool-test.cc:1284: Failure
Expected: cluster_->AssertNoCrashes() doesn't generate new fatal failures in the current thread.
  Actual: it does.
I20240502 02:01:31.432195 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 5982
I20240502 02:01:31.439666  6109 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 02:01:31.488962  6269 consensus_peers.cc:485] T 2cefdfa8851147999c1425c606f36b1f P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 02:01:31.568082  6136 consensus_peers.cc:462] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 9db17f84d37944098cbeb9e0d8456efb (127.25.109.134:42517): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.134:42517: connect: Connection refused (error 111)
W20240502 02:01:31.576170  6401 consensus_peers.cc:485] T 4eb91f5df851459b8ecf09cbe34fefa5 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 02:01:31.663688  6135 consensus_peers.cc:485] T d7922fd756344d14a336ff0c54afdf73 P a2c90ee440b44b708bccc2e2d5385cc0 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 02:01:31.825172 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 5982
W20240502 02:01:31.834472  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:01:31.883180  8204 connection.cc:535] server connection from 127.25.109.129:59983 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 02:01:31.883401  5905 connection.cc:535] server connection from 127.25.109.129:40579 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 02:01:31.883450  8204 connection.cc:535] client connection to 127.25.109.129:36227 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 02:01:31.883467  6402 connection.cc:535] client connection to 127.25.109.129:36227 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 02:01:31.883558  8204 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 02:01:31.884336  6401 connection.cc:535] server connection from 127.25.109.129:53907 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 02:01:31.884568 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 6119
W20240502 02:01:31.884824  6402 consensus_peers.cc:485] T f309b40931dd444091869119eab33bbc P 07cdb213c77c4ffda1f999b66b725399 -> Peer 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227): Couldn't send request to peer 218bc8c1f9ce46059898fc738179adf9. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:36227: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 02:01:31.885342  8204 consensus_peers.cc:485] T 89f7cf12d85442749e7d5b259ebca43f P 436822a397664c54b8d1b9eb3bc6cff4 -> Peer 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227): Couldn't send request to peer 218bc8c1f9ce46059898fc738179adf9. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:36227: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 02:01:31.890964  6243 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 02:01:31.922886  6402 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227): Couldn't send request to peer 218bc8c1f9ce46059898fc738179adf9. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:36227: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 02:01:31.960361  6269 consensus_peers.cc:485] T 9a5f486852554e2fa2d07f7b9d686ced P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 02:01:32.279454 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 6119
W20240502 02:01:32.283303  6270 consensus_peers.cc:485] T 0cecd1b14e474dee825fae040ce1b31e P 94ffd7aaf85b4e17881241b8396fa2f3 -> Peer 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227): Couldn't send request to peer 218bc8c1f9ce46059898fc738179adf9. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.129:36227: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 02:01:32.302795  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:01:32.334240  8204 connection.cc:535] server connection from 127.25.109.130:50235 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 02:01:32.334830 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 6253
I20240502 02:01:32.341012  6374 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 02:01:32.691327  6401 consensus_peers.cc:485] T 6984d638991442e3adaed1b24bee95e9 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 02:01:32.698307  6401 consensus_peers.cc:485] T 02b106ec57b94e44a29947a80a030a23 P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Couldn't send request to peer 6d34e50fb5dd4a1db58013d4863b8992. Status: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 02:01:32.708259  6401 consensus_peers.cc:462] T 4486b413e5c14ec6953e4895ee5797cf P 07cdb213c77c4ffda1f999b66b725399 -> Peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
I20240502 02:01:32.734413 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 6253
W20240502 02:01:32.791685  7936 connection.cc:535] server connection from 127.25.109.131:40629 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 02:01:32.792347 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 6385
I20240502 02:01:32.796547 28953 raft_consensus.cc:482] T 5be45a68cff24a75a03b2fcd4b7851f9 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 218bc8c1f9ce46059898fc738179adf9)
I20240502 02:01:32.796752 28953 raft_consensus.cc:504] T 5be45a68cff24a75a03b2fcd4b7851f9 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 02:01:32.797780 28953 leader_election.cc:288] T 5be45a68cff24a75a03b2fcd4b7851f9 P 436822a397664c54b8d1b9eb3bc6cff4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227), a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197)
I20240502 02:01:32.798923  6509 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 02:01:32.799327  8204 leader_election.cc:334] T 5be45a68cff24a75a03b2fcd4b7851f9 P 436822a397664c54b8d1b9eb3bc6cff4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227): Network error: Client connection negotiation failed: client connection to 127.25.109.129:36227: connect: Connection refused (error 111)
W20240502 02:01:32.799542  8205 leader_election.cc:334] T 5be45a68cff24a75a03b2fcd4b7851f9 P 436822a397664c54b8d1b9eb3bc6cff4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197): Network error: Client connection negotiation failed: client connection to 127.25.109.130:38197: connect: Connection refused (error 111)
I20240502 02:01:32.799646  8205 leader_election.cc:302] T 5be45a68cff24a75a03b2fcd4b7851f9 P 436822a397664c54b8d1b9eb3bc6cff4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 436822a397664c54b8d1b9eb3bc6cff4; no voters: 218bc8c1f9ce46059898fc738179adf9, a2c90ee440b44b708bccc2e2d5385cc0
I20240502 02:01:32.800217 28953 raft_consensus.cc:2726] T 5be45a68cff24a75a03b2fcd4b7851f9 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 02:01:33.190330 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 6385
I20240502 02:01:33.213001 28953 raft_consensus.cc:482] T 086509f1ee0a4a81b2f8194b50dbe159 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a2c90ee440b44b708bccc2e2d5385cc0)
I20240502 02:01:33.213245 28953 raft_consensus.cc:504] T 086509f1ee0a4a81b2f8194b50dbe159 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } }
I20240502 02:01:33.214156 28953 leader_election.cc:288] T 086509f1ee0a4a81b2f8194b50dbe159 P 436822a397664c54b8d1b9eb3bc6cff4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197), 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359)
W20240502 02:01:33.215449  8203 leader_election.cc:334] T 086509f1ee0a4a81b2f8194b50dbe159 P 436822a397664c54b8d1b9eb3bc6cff4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359): Network error: Client connection negotiation failed: client connection to 127.25.109.136:44359: connect: Connection refused (error 111)
W20240502 02:01:33.215649  8205 leader_election.cc:334] T 086509f1ee0a4a81b2f8194b50dbe159 P 436822a397664c54b8d1b9eb3bc6cff4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197): Network error: Client connection negotiation failed: client connection to 127.25.109.130:38197: connect: Connection refused (error 111)
I20240502 02:01:33.215790  8205 leader_election.cc:302] T 086509f1ee0a4a81b2f8194b50dbe159 P 436822a397664c54b8d1b9eb3bc6cff4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 436822a397664c54b8d1b9eb3bc6cff4; no voters: 6d34e50fb5dd4a1db58013d4863b8992, a2c90ee440b44b708bccc2e2d5385cc0
I20240502 02:01:33.216090 28953 raft_consensus.cc:2726] T 086509f1ee0a4a81b2f8194b50dbe159 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 02:01:33.245385  8202 connection.cc:535] server connection from 127.25.109.132:38165 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 02:01:33.245785  5907 connection.cc:535] server connection from 127.25.109.132:39769 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 02:01:33.245918 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 7918
I20240502 02:01:33.258462  8038 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 02:01:33.326200 28953 raft_consensus.cc:482] T 99e74a49b42745888058db67049dc682 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 218bc8c1f9ce46059898fc738179adf9)
I20240502 02:01:33.326408 28953 raft_consensus.cc:504] T 99e74a49b42745888058db67049dc682 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "07cdb213c77c4ffda1f999b66b725399" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 36867 } }
I20240502 02:01:33.326833 29889 raft_consensus.cc:482] T a398da35f18c4df394db82c5bf89aeb1 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 218bc8c1f9ce46059898fc738179adf9)
I20240502 02:01:33.327003 29889 raft_consensus.cc:504] T a398da35f18c4df394db82c5bf89aeb1 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "218bc8c1f9ce46059898fc738179adf9" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 36227 } } peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 02:01:33.327375 28953 leader_election.cc:288] T 99e74a49b42745888058db67049dc682 P 436822a397664c54b8d1b9eb3bc6cff4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227), 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867)
I20240502 02:01:33.328289 29889 leader_election.cc:288] T a398da35f18c4df394db82c5bf89aeb1 P 436822a397664c54b8d1b9eb3bc6cff4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227), 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245)
W20240502 02:01:33.328976  8204 leader_election.cc:334] T 99e74a49b42745888058db67049dc682 P 436822a397664c54b8d1b9eb3bc6cff4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227): Network error: Client connection negotiation failed: client connection to 127.25.109.129:36227: connect: Connection refused (error 111)
W20240502 02:01:33.329140  8202 leader_election.cc:334] T 99e74a49b42745888058db67049dc682 P 436822a397664c54b8d1b9eb3bc6cff4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 07cdb213c77c4ffda1f999b66b725399 (127.25.109.132:36867): Network error: Client connection negotiation failed: client connection to 127.25.109.132:36867: connect: Connection refused (error 111)
I20240502 02:01:33.329298  8202 leader_election.cc:302] T 99e74a49b42745888058db67049dc682 P 436822a397664c54b8d1b9eb3bc6cff4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 436822a397664c54b8d1b9eb3bc6cff4; no voters: 07cdb213c77c4ffda1f999b66b725399, 218bc8c1f9ce46059898fc738179adf9
I20240502 02:01:33.329588 29889 raft_consensus.cc:2726] T 99e74a49b42745888058db67049dc682 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 02:01:33.330493  8204 leader_election.cc:334] T a398da35f18c4df394db82c5bf89aeb1 P 436822a397664c54b8d1b9eb3bc6cff4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 218bc8c1f9ce46059898fc738179adf9 (127.25.109.129:36227): Network error: Client connection negotiation failed: client connection to 127.25.109.129:36227: connect: Connection refused (error 111)
W20240502 02:01:33.330683  8204 leader_election.cc:334] T a398da35f18c4df394db82c5bf89aeb1 P 436822a397664c54b8d1b9eb3bc6cff4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245): Network error: Client connection negotiation failed: client connection to 127.25.109.131:44245: connect: Connection refused (error 111)
I20240502 02:01:33.330776  8204 leader_election.cc:302] T a398da35f18c4df394db82c5bf89aeb1 P 436822a397664c54b8d1b9eb3bc6cff4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 436822a397664c54b8d1b9eb3bc6cff4; no voters: 218bc8c1f9ce46059898fc738179adf9, 94ffd7aaf85b4e17881241b8396fa2f3
I20240502 02:01:33.331005 29889 raft_consensus.cc:2726] T a398da35f18c4df394db82c5bf89aeb1 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 02:01:33.437815 29889 raft_consensus.cc:482] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 94ffd7aaf85b4e17881241b8396fa2f3)
I20240502 02:01:33.438015 29889 raft_consensus.cc:504] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "94ffd7aaf85b4e17881241b8396fa2f3" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 44245 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } } peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } }
I20240502 02:01:33.438732 29889 leader_election.cc:288] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245), a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197)
W20240502 02:01:33.440186  8205 leader_election.cc:334] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197): Network error: Client connection negotiation failed: client connection to 127.25.109.130:38197: connect: Connection refused (error 111)
W20240502 02:01:33.440339  8204 leader_election.cc:334] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 94ffd7aaf85b4e17881241b8396fa2f3 (127.25.109.131:44245): Network error: Client connection negotiation failed: client connection to 127.25.109.131:44245: connect: Connection refused (error 111)
I20240502 02:01:33.440446  8204 leader_election.cc:302] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 436822a397664c54b8d1b9eb3bc6cff4; no voters: 94ffd7aaf85b4e17881241b8396fa2f3, a2c90ee440b44b708bccc2e2d5385cc0
I20240502 02:01:33.440732 29889 raft_consensus.cc:2726] T afa5561f8c0e4c0fa9bbbd5da61348c7 P 436822a397664c54b8d1b9eb3bc6cff4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 02:01:33.611325 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 7918
I20240502 02:01:33.663390 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 8185
I20240502 02:01:33.669574  8308 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 02:01:34.016958 28953 raft_consensus.cc:482] T 73ed91df7ff04846991c95b30a86ea7e P 436822a397664c54b8d1b9eb3bc6cff4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a2c90ee440b44b708bccc2e2d5385cc0)
I20240502 02:01:34.017336 28953 raft_consensus.cc:504] T 73ed91df7ff04846991c95b30a86ea7e P 436822a397664c54b8d1b9eb3bc6cff4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2c90ee440b44b708bccc2e2d5385cc0" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38197 } } peers { permanent_uuid: "6d34e50fb5dd4a1db58013d4863b8992" member_type: VOTER last_known_addr { host: "127.25.109.136" port: 44359 } } peers { permanent_uuid: "436822a397664c54b8d1b9eb3bc6cff4" member_type: VOTER last_known_addr { host: "127.25.109.135" port: 40543 } }
I20240502 02:01:34.018111 28953 leader_election.cc:288] T 73ed91df7ff04846991c95b30a86ea7e P 436822a397664c54b8d1b9eb3bc6cff4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a2c90ee440b44b708bccc2e2d5385cc0 (127.25.109.130:38197), 6d34e50fb5dd4a1db58013d4863b8992 (127.25.109.136:44359)
I20240502 02:01:34.020795 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 8185
I20240502 02:01:34.075376 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 5890
I20240502 02:01:34.081307  5949 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 02:01:34.454972 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 5890
2024-05-02T02:01:34Z chronyd exiting
I20240502 02:01:34.529536 26038 test_util.cc:175] -----------------------------------------------
I20240502 02:01:34.529665 26038 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714614746325827-26038-0
[  FAILED  ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/0, where GetParam() = 4-byte object <00-00 00-00> (373597 ms)
[----------] 1 test from ConcurrentRebalancersTest (373598 ms total)

[----------] 1 test from RebalancerAndSingleReplicaTablets
[ RUN      ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/0
I20240502 02:01:34.532895 26038 test_util.cc:268] Using random seed: -74539793
I20240502 02:01:34.535111 26038 ts_itest-base.cc:112] Starting cluster with:
I20240502 02:01:34.535223 26038 ts_itest-base.cc:113] --------------
I20240502 02:01:34.535284 26038 ts_itest-base.cc:114] 4 tablet servers
I20240502 02:01:34.535368 26038 ts_itest-base.cc:115] 1 replicas per TS
I20240502 02:01:34.535426 26038 ts_itest-base.cc:116] --------------
2024-05-02T02:01:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T02:01:34Z Disabled control of system clock
I20240502 02:01:34.612428 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.190:34625
--webserver_interface=127.25.109.190
--webserver_port=0
--builtin_ntp_servers=127.25.109.148:36075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.190:34625
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false with env {}
W20240502 02:01:34.970870 29905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:01:34.971207 29905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 02:01:34.971315 29905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:01:34.979174 29905 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 02:01:34.979322 29905 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 02:01:34.979373 29905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 02:01:34.979431 29905 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 02:01:34.979480 29905 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 02:01:34.989125 29905 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:36075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.25.109.190:34625
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.190:34625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.25.109.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 02:01:34.990458 29905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 02:01:34.992216 29905 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 02:01:35.003365 29911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:01:35.003391 29910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:01:35.003692 29913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 02:01:35.004205 29905 server_base.cc:1034] running on GCE node
I20240502 02:01:35.004974 29905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 02:01:35.006385 29905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 02:01:35.007596 29905 hybrid_clock.cc:648] HybridClock initialized: now 1714615295007532 us; error 43 us; skew 500 ppm
I20240502 02:01:35.008002 29905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 02:01:35.010265 29905 webserver.cc:460] Webserver started at http://127.25.109.190:34051/ using document root <none> and password file <none>
I20240502 02:01:35.011031 29905 fs_manager.cc:359] Metadata directory not provided
I20240502 02:01:35.011147 29905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 02:01:35.011471 29905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 02:01:35.014325 29905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "cef66a04ddc4490fabb3151c4994b050"
format_stamp: "Formatted at 2024-05-02 02:01:35 on dist-test-slave-kj4b"
I20240502 02:01:35.015151 29905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "cef66a04ddc4490fabb3151c4994b050"
format_stamp: "Formatted at 2024-05-02 02:01:35 on dist-test-slave-kj4b"
I20240502 02:01:35.020515 29905 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
I20240502 02:01:35.023773 29920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:01:35.025110 29905 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240502 02:01:35.025269 29905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal
uuid: "cef66a04ddc4490fabb3151c4994b050"
format_stamp: "Formatted at 2024-05-02 02:01:35 on dist-test-slave-kj4b"
I20240502 02:01:35.025466 29905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 02:01:35.070792 29905 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 02:01:35.071748 29905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 02:01:35.072098 29905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 02:01:35.090039 29905 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.190:34625
I20240502 02:01:35.090060 29972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.190:34625 every 8 connection(s)
I20240502 02:01:35.091462 29905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 02:01:35.095237 29973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:35.098969 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 29905
I20240502 02:01:35.099206 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 02:01:35.106276 29973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cef66a04ddc4490fabb3151c4994b050: Bootstrap starting.
I20240502 02:01:35.109928 29973 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cef66a04ddc4490fabb3151c4994b050: Neither blocks nor log segments found. Creating new log.
I20240502 02:01:35.111333 29973 log.cc:824] T 00000000000000000000000000000000 P cef66a04ddc4490fabb3151c4994b050: Log is configured to *not* fsync() on all Append() calls
I20240502 02:01:35.114428 29973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cef66a04ddc4490fabb3151c4994b050: No bootstrap required, opened a new log
I20240502 02:01:35.120751 29973 raft_consensus.cc:348] T 00000000000000000000000000000000 P cef66a04ddc4490fabb3151c4994b050 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cef66a04ddc4490fabb3151c4994b050" member_type: VOTER last_known_addr { host: "127.25.109.190" port: 34625 } }
I20240502 02:01:35.121093 29973 raft_consensus.cc:374] T 00000000000000000000000000000000 P cef66a04ddc4490fabb3151c4994b050 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:01:35.121176 29973 raft_consensus.cc:729] T 00000000000000000000000000000000 P cef66a04ddc4490fabb3151c4994b050 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cef66a04ddc4490fabb3151c4994b050, State: Initialized, Role: FOLLOWER
I20240502 02:01:35.121748 29973 consensus_queue.cc:260] T 00000000000000000000000000000000 P cef66a04ddc4490fabb3151c4994b050 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cef66a04ddc4490fabb3151c4994b050" member_type: VOTER last_known_addr { host: "127.25.109.190" port: 34625 } }
I20240502 02:01:35.121965 29973 raft_consensus.cc:388] T 00000000000000000000000000000000 P cef66a04ddc4490fabb3151c4994b050 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 02:01:35.122115 29973 raft_consensus.cc:482] T 00000000000000000000000000000000 P cef66a04ddc4490fabb3151c4994b050 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 02:01:35.122303 29973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cef66a04ddc4490fabb3151c4994b050 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:01:35.124436 29973 raft_consensus.cc:504] T 00000000000000000000000000000000 P cef66a04ddc4490fabb3151c4994b050 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cef66a04ddc4490fabb3151c4994b050" member_type: VOTER last_known_addr { host: "127.25.109.190" port: 34625 } }
I20240502 02:01:35.124897 29973 leader_election.cc:302] T 00000000000000000000000000000000 P cef66a04ddc4490fabb3151c4994b050 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cef66a04ddc4490fabb3151c4994b050; no voters: 
I20240502 02:01:35.125434 29973 leader_election.cc:288] T 00000000000000000000000000000000 P cef66a04ddc4490fabb3151c4994b050 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 02:01:35.125607 29978 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cef66a04ddc4490fabb3151c4994b050 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:01:35.126209 29978 raft_consensus.cc:686] T 00000000000000000000000000000000 P cef66a04ddc4490fabb3151c4994b050 [term 1 LEADER]: Becoming Leader. State: Replica: cef66a04ddc4490fabb3151c4994b050, State: Running, Role: LEADER
I20240502 02:01:35.126719 29978 consensus_queue.cc:237] T 00000000000000000000000000000000 P cef66a04ddc4490fabb3151c4994b050 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cef66a04ddc4490fabb3151c4994b050" member_type: VOTER last_known_addr { host: "127.25.109.190" port: 34625 } }
I20240502 02:01:35.127355 29973 sys_catalog.cc:564] T 00000000000000000000000000000000 P cef66a04ddc4490fabb3151c4994b050 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 02:01:35.129446 29980 sys_catalog.cc:455] T 00000000000000000000000000000000 P cef66a04ddc4490fabb3151c4994b050 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cef66a04ddc4490fabb3151c4994b050. Latest consensus state: current_term: 1 leader_uuid: "cef66a04ddc4490fabb3151c4994b050" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cef66a04ddc4490fabb3151c4994b050" member_type: VOTER last_known_addr { host: "127.25.109.190" port: 34625 } } }
I20240502 02:01:35.129535 29979 sys_catalog.cc:455] T 00000000000000000000000000000000 P cef66a04ddc4490fabb3151c4994b050 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cef66a04ddc4490fabb3151c4994b050" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cef66a04ddc4490fabb3151c4994b050" member_type: VOTER last_known_addr { host: "127.25.109.190" port: 34625 } } }
I20240502 02:01:35.129812 29980 sys_catalog.cc:458] T 00000000000000000000000000000000 P cef66a04ddc4490fabb3151c4994b050 [sys.catalog]: This master's current role is: LEADER
I20240502 02:01:35.129859 29979 sys_catalog.cc:458] T 00000000000000000000000000000000 P cef66a04ddc4490fabb3151c4994b050 [sys.catalog]: This master's current role is: LEADER
I20240502 02:01:35.130813 29984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 02:01:35.136698 29984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 02:01:35.146333 29984 catalog_manager.cc:1347] Generated new cluster ID: 072f64e0ce764a83949e05f36339a888
I20240502 02:01:35.146462 29984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 02:01:35.155750 29984 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 02:01:35.157624 29984 catalog_manager.cc:1504] Loading token signing keys...
I20240502 02:01:35.170709 29984 catalog_manager.cc:5857] T 00000000000000000000000000000000 P cef66a04ddc4490fabb3151c4994b050: Generated new TSK 0
I20240502 02:01:35.171891 29984 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 02:01:35.178218 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.129:0
--local_ip_for_outbound_sockets=127.25.109.129
--webserver_interface=127.25.109.129
--webserver_port=0
--tserver_master_addrs=127.25.109.190:34625
--builtin_ntp_servers=127.25.109.148:36075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 02:01:35.535544 29998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:01:35.535842 29998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 02:01:35.536008 29998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:01:35.543795 29998 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 02:01:35.543964 29998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 02:01:35.544117 29998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.129
I20240502 02:01:35.553421 29998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:36075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.25.109.129
--webserver_port=0
--tserver_master_addrs=127.25.109.190:34625
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.129
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 02:01:35.554771 29998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 02:01:35.556586 29998 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 02:01:35.568571 30003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:01:35.568799 30006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:01:35.569173 30004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 02:01:35.569388 29998 server_base.cc:1034] running on GCE node
I20240502 02:01:35.570060 29998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 02:01:35.571043 29998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 02:01:35.572244 29998 hybrid_clock.cc:648] HybridClock initialized: now 1714615295572191 us; error 48 us; skew 500 ppm
I20240502 02:01:35.572588 29998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 02:01:35.574942 29998 webserver.cc:460] Webserver started at http://127.25.109.129:34047/ using document root <none> and password file <none>
I20240502 02:01:35.575701 29998 fs_manager.cc:359] Metadata directory not provided
I20240502 02:01:35.575825 29998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 02:01:35.576159 29998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 02:01:35.578886 29998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "dadc9384397645eebe8a77c70a2b883a"
format_stamp: "Formatted at 2024-05-02 02:01:35 on dist-test-slave-kj4b"
I20240502 02:01:35.579700 29998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "dadc9384397645eebe8a77c70a2b883a"
format_stamp: "Formatted at 2024-05-02 02:01:35 on dist-test-slave-kj4b"
I20240502 02:01:35.585180 29998 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.001s
I20240502 02:01:35.588829 30012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:01:35.590271 29998 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20240502 02:01:35.590451 29998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "dadc9384397645eebe8a77c70a2b883a"
format_stamp: "Formatted at 2024-05-02 02:01:35 on dist-test-slave-kj4b"
I20240502 02:01:35.590665 29998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 02:01:35.617873 29998 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 02:01:35.618803 29998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 02:01:35.619107 29998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 02:01:35.620123 29998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 02:01:35.621655 29998 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 02:01:35.621754 29998 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:01:35.621861 29998 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 02:01:35.621938 29998 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:01:35.653110 29998 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.129:41377
I20240502 02:01:35.653183 30128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.129:41377 every 8 connection(s)
I20240502 02:01:35.654510 29998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 02:01:35.655066 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 29998
I20240502 02:01:35.655288 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 02:01:35.659345 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.130:0
--local_ip_for_outbound_sockets=127.25.109.130
--webserver_interface=127.25.109.130
--webserver_port=0
--tserver_master_addrs=127.25.109.190:34625
--builtin_ntp_servers=127.25.109.148:36075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 02:01:35.669503 30129 heartbeater.cc:347] Connected to a master server at 127.25.109.190:34625
I20240502 02:01:35.669878 30129 heartbeater.cc:464] Registering TS with master...
I20240502 02:01:35.670732 30129 heartbeater.cc:511] Master 127.25.109.190:34625 requested a full tablet report, sending...
I20240502 02:01:35.673085 29938 ts_manager.cc:194] Registered new tserver with Master: dadc9384397645eebe8a77c70a2b883a (127.25.109.129:41377)
I20240502 02:01:35.675150 29938 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.129:38235
W20240502 02:01:36.018350 30133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:01:36.018671 30133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 02:01:36.018807 30133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:01:36.026511 30133 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 02:01:36.026664 30133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 02:01:36.026827 30133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.130
I20240502 02:01:36.036547 30133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:36075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.25.109.130
--webserver_port=0
--tserver_master_addrs=127.25.109.190:34625
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.130
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 02:01:36.037920 30133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 02:01:36.039808 30133 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 02:01:36.051837 30139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:01:36.052254 30138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:01:36.052675 30141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 02:01:36.053157 30133 server_base.cc:1034] running on GCE node
I20240502 02:01:36.053797 30133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 02:01:36.054807 30133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 02:01:36.056003 30133 hybrid_clock.cc:648] HybridClock initialized: now 1714615296055938 us; error 47 us; skew 500 ppm
I20240502 02:01:36.056411 30133 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 02:01:36.058879 30133 webserver.cc:460] Webserver started at http://127.25.109.130:37073/ using document root <none> and password file <none>
I20240502 02:01:36.059685 30133 fs_manager.cc:359] Metadata directory not provided
I20240502 02:01:36.059815 30133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 02:01:36.060168 30133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 02:01:36.062952 30133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "a49cde355f1e4825989af6cb6c874aee"
format_stamp: "Formatted at 2024-05-02 02:01:36 on dist-test-slave-kj4b"
I20240502 02:01:36.063767 30133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "a49cde355f1e4825989af6cb6c874aee"
format_stamp: "Formatted at 2024-05-02 02:01:36 on dist-test-slave-kj4b"
I20240502 02:01:36.069482 30133 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20240502 02:01:36.073186 30147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:01:36.074669 30133 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240502 02:01:36.074889 30133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "a49cde355f1e4825989af6cb6c874aee"
format_stamp: "Formatted at 2024-05-02 02:01:36 on dist-test-slave-kj4b"
I20240502 02:01:36.075119 30133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 02:01:36.093675 30133 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 02:01:36.094699 30133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 02:01:36.095013 30133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 02:01:36.096107 30133 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 02:01:36.097867 30133 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 02:01:36.097980 30133 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:01:36.098093 30133 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 02:01:36.098160 30133 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:01:36.129246 30133 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.130:33089
I20240502 02:01:36.129297 30262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.130:33089 every 8 connection(s)
I20240502 02:01:36.130651 30133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 02:01:36.138103 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 30133
I20240502 02:01:36.138307 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 02:01:36.141628 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.131:0
--local_ip_for_outbound_sockets=127.25.109.131
--webserver_interface=127.25.109.131
--webserver_port=0
--tserver_master_addrs=127.25.109.190:34625
--builtin_ntp_servers=127.25.109.148:36075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 02:01:36.143805 30263 heartbeater.cc:347] Connected to a master server at 127.25.109.190:34625
I20240502 02:01:36.144140 30263 heartbeater.cc:464] Registering TS with master...
I20240502 02:01:36.144892 30263 heartbeater.cc:511] Master 127.25.109.190:34625 requested a full tablet report, sending...
I20240502 02:01:36.146307 29938 ts_manager.cc:194] Registered new tserver with Master: a49cde355f1e4825989af6cb6c874aee (127.25.109.130:33089)
I20240502 02:01:36.147250 29938 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.130:37061
W20240502 02:01:36.493136 30266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:01:36.493458 30266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 02:01:36.493551 30266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:01:36.501344 30266 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 02:01:36.501500 30266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 02:01:36.501631 30266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.131
I20240502 02:01:36.511279 30266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:36075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.25.109.131
--webserver_port=0
--tserver_master_addrs=127.25.109.190:34625
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.131
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 02:01:36.512696 30266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 02:01:36.514504 30266 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 02:01:36.526935 30274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:01:36.527010 30271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:01:36.527449 30272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 02:01:36.528019 30266 server_base.cc:1034] running on GCE node
I20240502 02:01:36.528628 30266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 02:01:36.529695 30266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 02:01:36.530901 30266 hybrid_clock.cc:648] HybridClock initialized: now 1714615296530851 us; error 37 us; skew 500 ppm
I20240502 02:01:36.531239 30266 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 02:01:36.533767 30266 webserver.cc:460] Webserver started at http://127.25.109.131:33085/ using document root <none> and password file <none>
I20240502 02:01:36.534538 30266 fs_manager.cc:359] Metadata directory not provided
I20240502 02:01:36.534667 30266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 02:01:36.534976 30266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 02:01:36.537815 30266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "59daeed1fd4d4db5b7b41e678bb22c21"
format_stamp: "Formatted at 2024-05-02 02:01:36 on dist-test-slave-kj4b"
I20240502 02:01:36.538671 30266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "59daeed1fd4d4db5b7b41e678bb22c21"
format_stamp: "Formatted at 2024-05-02 02:01:36 on dist-test-slave-kj4b"
I20240502 02:01:36.544456 30266 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.005s
I20240502 02:01:36.548256 30280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:01:36.549726 30266 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240502 02:01:36.549914 30266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "59daeed1fd4d4db5b7b41e678bb22c21"
format_stamp: "Formatted at 2024-05-02 02:01:36 on dist-test-slave-kj4b"
I20240502 02:01:36.550148 30266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 02:01:36.566128 30266 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 02:01:36.567059 30266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 02:01:36.567343 30266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 02:01:36.568434 30266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 02:01:36.570320 30266 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 02:01:36.570438 30266 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:01:36.570554 30266 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 02:01:36.570623 30266 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:01:36.601696 30266 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.131:38831
I20240502 02:01:36.601789 30395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.131:38831 every 8 connection(s)
I20240502 02:01:36.603063 30266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 02:01:36.610720 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 30266
I20240502 02:01:36.610908 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 02:01:36.614516 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.132:0
--local_ip_for_outbound_sockets=127.25.109.132
--webserver_interface=127.25.109.132
--webserver_port=0
--tserver_master_addrs=127.25.109.190:34625
--builtin_ntp_servers=127.25.109.148:36075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 02:01:36.615732 30396 heartbeater.cc:347] Connected to a master server at 127.25.109.190:34625
I20240502 02:01:36.616066 30396 heartbeater.cc:464] Registering TS with master...
I20240502 02:01:36.616823 30396 heartbeater.cc:511] Master 127.25.109.190:34625 requested a full tablet report, sending...
I20240502 02:01:36.618219 29938 ts_manager.cc:194] Registered new tserver with Master: 59daeed1fd4d4db5b7b41e678bb22c21 (127.25.109.131:38831)
I20240502 02:01:36.619009 29938 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.131:54679
I20240502 02:01:36.678200 30129 heartbeater.cc:503] Master 127.25.109.190:34625 was elected leader, sending a full tablet report...
W20240502 02:01:36.964793 30399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:01:36.965132 30399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 02:01:36.965274 30399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:01:36.973196 30399 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 02:01:36.973356 30399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 02:01:36.973518 30399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.132
I20240502 02:01:36.982970 30399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:36075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.25.109.132
--webserver_port=0
--tserver_master_addrs=127.25.109.190:34625
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.132
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 02:01:36.984356 30399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 02:01:36.986121 30399 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 02:01:36.998610 30405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:01:36.998731 30407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:01:36.999037 30404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 02:01:36.999531 30399 server_base.cc:1034] running on GCE node
I20240502 02:01:37.000191 30399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 02:01:37.001152 30399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 02:01:37.002351 30399 hybrid_clock.cc:648] HybridClock initialized: now 1714615297002290 us; error 57 us; skew 500 ppm
I20240502 02:01:37.002692 30399 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 02:01:37.004940 30399 webserver.cc:460] Webserver started at http://127.25.109.132:39215/ using document root <none> and password file <none>
I20240502 02:01:37.005695 30399 fs_manager.cc:359] Metadata directory not provided
I20240502 02:01:37.005820 30399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 02:01:37.006135 30399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 02:01:37.008900 30399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "4180f0eefa0642ad9ab53b9dd07b8cf8"
format_stamp: "Formatted at 2024-05-02 02:01:37 on dist-test-slave-kj4b"
I20240502 02:01:37.009680 30399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "4180f0eefa0642ad9ab53b9dd07b8cf8"
format_stamp: "Formatted at 2024-05-02 02:01:37 on dist-test-slave-kj4b"
I20240502 02:01:37.015177 30399 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.001s
I20240502 02:01:37.018833 30413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:01:37.020265 30399 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 02:01:37.020437 30399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "4180f0eefa0642ad9ab53b9dd07b8cf8"
format_stamp: "Formatted at 2024-05-02 02:01:37 on dist-test-slave-kj4b"
I20240502 02:01:37.020644 30399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 02:01:37.048525 30399 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 02:01:37.049528 30399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 02:01:37.049804 30399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 02:01:37.050824 30399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 02:01:37.052448 30399 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 02:01:37.052549 30399 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:01:37.052664 30399 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 02:01:37.052732 30399 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:01:37.083698 30399 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.132:45285
I20240502 02:01:37.083770 30528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.132:45285 every 8 connection(s)
I20240502 02:01:37.085086 30399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 02:01:37.091217 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 30399
I20240502 02:01:37.091423 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 02:01:37.099749 30529 heartbeater.cc:347] Connected to a master server at 127.25.109.190:34625
I20240502 02:01:37.100106 30529 heartbeater.cc:464] Registering TS with master...
I20240502 02:01:37.100891 30529 heartbeater.cc:511] Master 127.25.109.190:34625 requested a full tablet report, sending...
I20240502 02:01:37.102283 29938 ts_manager.cc:194] Registered new tserver with Master: 4180f0eefa0642ad9ab53b9dd07b8cf8 (127.25.109.132:45285)
I20240502 02:01:37.103117 29938 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.132:47741
I20240502 02:01:37.107997 26038 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240502 02:01:37.123931 29938 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43150:
name: "TestTable"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
owner: "alice"
I20240502 02:01:37.149839 30263 heartbeater.cc:503] Master 127.25.109.190:34625 was elected leader, sending a full tablet report...
I20240502 02:01:37.153151 30197 tablet_service.cc:1467] Processing CreateTablet for tablet eebfabb6d8b04aabb2c34cfbb26fe0b9 (DEFAULT_TABLE table=TestTable [id=12b43007fb4c4bcab54d8aad0408ede2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 02:01:37.154618 30197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eebfabb6d8b04aabb2c34cfbb26fe0b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:37.165179 30544 tablet_bootstrap.cc:492] T eebfabb6d8b04aabb2c34cfbb26fe0b9 P a49cde355f1e4825989af6cb6c874aee: Bootstrap starting.
I20240502 02:01:37.168524 30544 tablet_bootstrap.cc:654] T eebfabb6d8b04aabb2c34cfbb26fe0b9 P a49cde355f1e4825989af6cb6c874aee: Neither blocks nor log segments found. Creating new log.
I20240502 02:01:37.169714 30544 log.cc:824] T eebfabb6d8b04aabb2c34cfbb26fe0b9 P a49cde355f1e4825989af6cb6c874aee: Log is configured to *not* fsync() on all Append() calls
I20240502 02:01:37.172703 30544 tablet_bootstrap.cc:492] T eebfabb6d8b04aabb2c34cfbb26fe0b9 P a49cde355f1e4825989af6cb6c874aee: No bootstrap required, opened a new log
I20240502 02:01:37.173022 30544 ts_tablet_manager.cc:1367] T eebfabb6d8b04aabb2c34cfbb26fe0b9 P a49cde355f1e4825989af6cb6c874aee: Time spent bootstrapping tablet: real 0.008s	user 0.007s	sys 0.000s
I20240502 02:01:37.179656 30544 raft_consensus.cc:348] T eebfabb6d8b04aabb2c34cfbb26fe0b9 P a49cde355f1e4825989af6cb6c874aee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49cde355f1e4825989af6cb6c874aee" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 33089 } }
I20240502 02:01:37.180112 30544 raft_consensus.cc:374] T eebfabb6d8b04aabb2c34cfbb26fe0b9 P a49cde355f1e4825989af6cb6c874aee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:01:37.180233 30544 raft_consensus.cc:729] T eebfabb6d8b04aabb2c34cfbb26fe0b9 P a49cde355f1e4825989af6cb6c874aee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a49cde355f1e4825989af6cb6c874aee, State: Initialized, Role: FOLLOWER
I20240502 02:01:37.180900 30544 consensus_queue.cc:260] T eebfabb6d8b04aabb2c34cfbb26fe0b9 P a49cde355f1e4825989af6cb6c874aee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49cde355f1e4825989af6cb6c874aee" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 33089 } }
I20240502 02:01:37.181136 30544 raft_consensus.cc:388] T eebfabb6d8b04aabb2c34cfbb26fe0b9 P a49cde355f1e4825989af6cb6c874aee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 02:01:37.181283 30544 raft_consensus.cc:482] T eebfabb6d8b04aabb2c34cfbb26fe0b9 P a49cde355f1e4825989af6cb6c874aee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 02:01:37.181442 30544 raft_consensus.cc:3037] T eebfabb6d8b04aabb2c34cfbb26fe0b9 P a49cde355f1e4825989af6cb6c874aee [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:01:37.183828 30544 raft_consensus.cc:504] T eebfabb6d8b04aabb2c34cfbb26fe0b9 P a49cde355f1e4825989af6cb6c874aee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49cde355f1e4825989af6cb6c874aee" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 33089 } }
I20240502 02:01:37.184487 30544 leader_election.cc:302] T eebfabb6d8b04aabb2c34cfbb26fe0b9 P a49cde355f1e4825989af6cb6c874aee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a49cde355f1e4825989af6cb6c874aee; no voters: 
I20240502 02:01:37.185127 30544 leader_election.cc:288] T eebfabb6d8b04aabb2c34cfbb26fe0b9 P a49cde355f1e4825989af6cb6c874aee [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 02:01:37.185297 30546 raft_consensus.cc:2781] T eebfabb6d8b04aabb2c34cfbb26fe0b9 P a49cde355f1e4825989af6cb6c874aee [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:01:37.185932 30546 raft_consensus.cc:686] T eebfabb6d8b04aabb2c34cfbb26fe0b9 P a49cde355f1e4825989af6cb6c874aee [term 1 LEADER]: Becoming Leader. State: Replica: a49cde355f1e4825989af6cb6c874aee, State: Running, Role: LEADER
I20240502 02:01:37.186782 30544 ts_tablet_manager.cc:1398] T eebfabb6d8b04aabb2c34cfbb26fe0b9 P a49cde355f1e4825989af6cb6c874aee: Time spent starting tablet: real 0.014s	user 0.005s	sys 0.008s
I20240502 02:01:37.186961 30546 consensus_queue.cc:237] T eebfabb6d8b04aabb2c34cfbb26fe0b9 P a49cde355f1e4825989af6cb6c874aee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49cde355f1e4825989af6cb6c874aee" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 33089 } }
I20240502 02:01:37.192361 29938 catalog_manager.cc:5484] T eebfabb6d8b04aabb2c34cfbb26fe0b9 P a49cde355f1e4825989af6cb6c874aee reported cstate change: term changed from 0 to 1, leader changed from <none> to a49cde355f1e4825989af6cb6c874aee (127.25.109.130). New cstate: current_term: 1 leader_uuid: "a49cde355f1e4825989af6cb6c874aee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49cde355f1e4825989af6cb6c874aee" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 33089 } } }
I20240502 02:01:37.211804 26038 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240502 02:01:37.213970 26038 ts_itest-base.cc:233] Waiting for 0 tablets on tserver dadc9384397645eebe8a77c70a2b883a to finish bootstrapping
I20240502 02:01:37.223167 26038 ts_itest-base.cc:233] Waiting for 1 tablets on tserver a49cde355f1e4825989af6cb6c874aee to finish bootstrapping
I20240502 02:01:37.231079 26038 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 59daeed1fd4d4db5b7b41e678bb22c21 to finish bootstrapping
I20240502 02:01:37.240196 26038 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 4180f0eefa0642ad9ab53b9dd07b8cf8 to finish bootstrapping
I20240502 02:01:37.249256 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 30266
I20240502 02:01:37.250396 30390 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 02:01:37.331298 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 30266
I20240502 02:01:37.346405 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 30399
I20240502 02:01:37.347522 30523 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 02:01:37.429427 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 30399
I20240502 02:01:41.196646 29938 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43150:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 02:01:41.216638 30060 tablet_service.cc:1467] Processing CreateTablet for tablet 8fcc05fb4cd949ae96776fe4bfb6317b (DEFAULT_TABLE table=rebalance_test_table_0 [id=f6bcdf7b88f44481bdcf022fb0141d56]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 02:01:41.216590 30058 tablet_service.cc:1467] Processing CreateTablet for tablet 09c1ed40361246a39b34cabe0fa9ec65 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f6bcdf7b88f44481bdcf022fb0141d56]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 02:01:41.216542 30059 tablet_service.cc:1467] Processing CreateTablet for tablet 89a27f63639244129d249d42ae4bb291 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f6bcdf7b88f44481bdcf022fb0141d56]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 02:01:41.218140 30059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89a27f63639244129d249d42ae4bb291. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:41.218611 30058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09c1ed40361246a39b34cabe0fa9ec65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:41.218860 30060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fcc05fb4cd949ae96776fe4bfb6317b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:41.236500 30563 tablet_bootstrap.cc:492] T 8fcc05fb4cd949ae96776fe4bfb6317b P dadc9384397645eebe8a77c70a2b883a: Bootstrap starting.
I20240502 02:01:41.239749 30563 tablet_bootstrap.cc:654] T 8fcc05fb4cd949ae96776fe4bfb6317b P dadc9384397645eebe8a77c70a2b883a: Neither blocks nor log segments found. Creating new log.
I20240502 02:01:41.240839 30563 log.cc:824] T 8fcc05fb4cd949ae96776fe4bfb6317b P dadc9384397645eebe8a77c70a2b883a: Log is configured to *not* fsync() on all Append() calls
I20240502 02:01:41.243252 30563 tablet_bootstrap.cc:492] T 8fcc05fb4cd949ae96776fe4bfb6317b P dadc9384397645eebe8a77c70a2b883a: No bootstrap required, opened a new log
I20240502 02:01:41.243510 30563 ts_tablet_manager.cc:1367] T 8fcc05fb4cd949ae96776fe4bfb6317b P dadc9384397645eebe8a77c70a2b883a: Time spent bootstrapping tablet: real 0.007s	user 0.005s	sys 0.000s
I20240502 02:01:41.250105 30563 raft_consensus.cc:348] T 8fcc05fb4cd949ae96776fe4bfb6317b P dadc9384397645eebe8a77c70a2b883a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dadc9384397645eebe8a77c70a2b883a" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 41377 } }
I20240502 02:01:41.250514 30563 raft_consensus.cc:374] T 8fcc05fb4cd949ae96776fe4bfb6317b P dadc9384397645eebe8a77c70a2b883a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:01:41.250620 30563 raft_consensus.cc:729] T 8fcc05fb4cd949ae96776fe4bfb6317b P dadc9384397645eebe8a77c70a2b883a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dadc9384397645eebe8a77c70a2b883a, State: Initialized, Role: FOLLOWER
I20240502 02:01:41.251257 30563 consensus_queue.cc:260] T 8fcc05fb4cd949ae96776fe4bfb6317b P dadc9384397645eebe8a77c70a2b883a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dadc9384397645eebe8a77c70a2b883a" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 41377 } }
I20240502 02:01:41.251507 30563 raft_consensus.cc:388] T 8fcc05fb4cd949ae96776fe4bfb6317b P dadc9384397645eebe8a77c70a2b883a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 02:01:41.251641 30563 raft_consensus.cc:482] T 8fcc05fb4cd949ae96776fe4bfb6317b P dadc9384397645eebe8a77c70a2b883a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 02:01:41.251809 30563 raft_consensus.cc:3037] T 8fcc05fb4cd949ae96776fe4bfb6317b P dadc9384397645eebe8a77c70a2b883a [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:01:41.253934 30563 raft_consensus.cc:504] T 8fcc05fb4cd949ae96776fe4bfb6317b P dadc9384397645eebe8a77c70a2b883a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dadc9384397645eebe8a77c70a2b883a" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 41377 } }
I20240502 02:01:41.254485 30563 leader_election.cc:302] T 8fcc05fb4cd949ae96776fe4bfb6317b P dadc9384397645eebe8a77c70a2b883a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dadc9384397645eebe8a77c70a2b883a; no voters: 
I20240502 02:01:41.255103 30563 leader_election.cc:288] T 8fcc05fb4cd949ae96776fe4bfb6317b P dadc9384397645eebe8a77c70a2b883a [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 02:01:41.255282 30565 raft_consensus.cc:2781] T 8fcc05fb4cd949ae96776fe4bfb6317b P dadc9384397645eebe8a77c70a2b883a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:01:41.255997 30565 raft_consensus.cc:686] T 8fcc05fb4cd949ae96776fe4bfb6317b P dadc9384397645eebe8a77c70a2b883a [term 1 LEADER]: Becoming Leader. State: Replica: dadc9384397645eebe8a77c70a2b883a, State: Running, Role: LEADER
I20240502 02:01:41.256680 30565 consensus_queue.cc:237] T 8fcc05fb4cd949ae96776fe4bfb6317b P dadc9384397645eebe8a77c70a2b883a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dadc9384397645eebe8a77c70a2b883a" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 41377 } }
I20240502 02:01:41.256919 30563 ts_tablet_manager.cc:1398] T 8fcc05fb4cd949ae96776fe4bfb6317b P dadc9384397645eebe8a77c70a2b883a: Time spent starting tablet: real 0.013s	user 0.010s	sys 0.004s
I20240502 02:01:41.257298 30563 tablet_bootstrap.cc:492] T 89a27f63639244129d249d42ae4bb291 P dadc9384397645eebe8a77c70a2b883a: Bootstrap starting.
I20240502 02:01:41.258917 30563 tablet_bootstrap.cc:654] T 89a27f63639244129d249d42ae4bb291 P dadc9384397645eebe8a77c70a2b883a: Neither blocks nor log segments found. Creating new log.
I20240502 02:01:41.261041 30563 tablet_bootstrap.cc:492] T 89a27f63639244129d249d42ae4bb291 P dadc9384397645eebe8a77c70a2b883a: No bootstrap required, opened a new log
I20240502 02:01:41.261274 30563 ts_tablet_manager.cc:1367] T 89a27f63639244129d249d42ae4bb291 P dadc9384397645eebe8a77c70a2b883a: Time spent bootstrapping tablet: real 0.004s	user 0.001s	sys 0.003s
I20240502 02:01:41.261852 29938 catalog_manager.cc:5484] T 8fcc05fb4cd949ae96776fe4bfb6317b P dadc9384397645eebe8a77c70a2b883a reported cstate change: term changed from 0 to 1, leader changed from <none> to dadc9384397645eebe8a77c70a2b883a (127.25.109.129). New cstate: current_term: 1 leader_uuid: "dadc9384397645eebe8a77c70a2b883a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dadc9384397645eebe8a77c70a2b883a" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 41377 } } }
I20240502 02:01:41.262300 30563 raft_consensus.cc:348] T 89a27f63639244129d249d42ae4bb291 P dadc9384397645eebe8a77c70a2b883a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dadc9384397645eebe8a77c70a2b883a" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 41377 } }
I20240502 02:01:41.262451 30563 raft_consensus.cc:374] T 89a27f63639244129d249d42ae4bb291 P dadc9384397645eebe8a77c70a2b883a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:01:41.262537 30563 raft_consensus.cc:729] T 89a27f63639244129d249d42ae4bb291 P dadc9384397645eebe8a77c70a2b883a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dadc9384397645eebe8a77c70a2b883a, State: Initialized, Role: FOLLOWER
I20240502 02:01:41.262784 30563 consensus_queue.cc:260] T 89a27f63639244129d249d42ae4bb291 P dadc9384397645eebe8a77c70a2b883a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dadc9384397645eebe8a77c70a2b883a" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 41377 } }
I20240502 02:01:41.263013 30563 raft_consensus.cc:388] T 89a27f63639244129d249d42ae4bb291 P dadc9384397645eebe8a77c70a2b883a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 02:01:41.263103 30563 raft_consensus.cc:482] T 89a27f63639244129d249d42ae4bb291 P dadc9384397645eebe8a77c70a2b883a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 02:01:41.263216 30563 raft_consensus.cc:3037] T 89a27f63639244129d249d42ae4bb291 P dadc9384397645eebe8a77c70a2b883a [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:01:41.265627 30563 raft_consensus.cc:504] T 89a27f63639244129d249d42ae4bb291 P dadc9384397645eebe8a77c70a2b883a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dadc9384397645eebe8a77c70a2b883a" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 41377 } }
I20240502 02:01:41.265920 30563 leader_election.cc:302] T 89a27f63639244129d249d42ae4bb291 P dadc9384397645eebe8a77c70a2b883a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dadc9384397645eebe8a77c70a2b883a; no voters: 
I20240502 02:01:41.266132 30563 leader_election.cc:288] T 89a27f63639244129d249d42ae4bb291 P dadc9384397645eebe8a77c70a2b883a [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 02:01:41.266213 30565 raft_consensus.cc:2781] T 89a27f63639244129d249d42ae4bb291 P dadc9384397645eebe8a77c70a2b883a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:01:41.266496 30565 raft_consensus.cc:686] T 89a27f63639244129d249d42ae4bb291 P dadc9384397645eebe8a77c70a2b883a [term 1 LEADER]: Becoming Leader. State: Replica: dadc9384397645eebe8a77c70a2b883a, State: Running, Role: LEADER
I20240502 02:01:41.266613 30563 ts_tablet_manager.cc:1398] T 89a27f63639244129d249d42ae4bb291 P dadc9384397645eebe8a77c70a2b883a: Time spent starting tablet: real 0.005s	user 0.000s	sys 0.003s
I20240502 02:01:41.266762 30565 consensus_queue.cc:237] T 89a27f63639244129d249d42ae4bb291 P dadc9384397645eebe8a77c70a2b883a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dadc9384397645eebe8a77c70a2b883a" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 41377 } }
I20240502 02:01:41.266913 30563 tablet_bootstrap.cc:492] T 09c1ed40361246a39b34cabe0fa9ec65 P dadc9384397645eebe8a77c70a2b883a: Bootstrap starting.
I20240502 02:01:41.268785 30563 tablet_bootstrap.cc:654] T 09c1ed40361246a39b34cabe0fa9ec65 P dadc9384397645eebe8a77c70a2b883a: Neither blocks nor log segments found. Creating new log.
I20240502 02:01:41.269842 29938 catalog_manager.cc:5484] T 89a27f63639244129d249d42ae4bb291 P dadc9384397645eebe8a77c70a2b883a reported cstate change: term changed from 0 to 1, leader changed from <none> to dadc9384397645eebe8a77c70a2b883a (127.25.109.129). New cstate: current_term: 1 leader_uuid: "dadc9384397645eebe8a77c70a2b883a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dadc9384397645eebe8a77c70a2b883a" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 41377 } } }
I20240502 02:01:41.270695 30563 tablet_bootstrap.cc:492] T 09c1ed40361246a39b34cabe0fa9ec65 P dadc9384397645eebe8a77c70a2b883a: No bootstrap required, opened a new log
I20240502 02:01:41.270851 30563 ts_tablet_manager.cc:1367] T 09c1ed40361246a39b34cabe0fa9ec65 P dadc9384397645eebe8a77c70a2b883a: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240502 02:01:41.271447 30563 raft_consensus.cc:348] T 09c1ed40361246a39b34cabe0fa9ec65 P dadc9384397645eebe8a77c70a2b883a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dadc9384397645eebe8a77c70a2b883a" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 41377 } }
I20240502 02:01:41.271728 30563 raft_consensus.cc:374] T 09c1ed40361246a39b34cabe0fa9ec65 P dadc9384397645eebe8a77c70a2b883a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:01:41.271848 30563 raft_consensus.cc:729] T 09c1ed40361246a39b34cabe0fa9ec65 P dadc9384397645eebe8a77c70a2b883a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dadc9384397645eebe8a77c70a2b883a, State: Initialized, Role: FOLLOWER
I20240502 02:01:41.272135 30563 consensus_queue.cc:260] T 09c1ed40361246a39b34cabe0fa9ec65 P dadc9384397645eebe8a77c70a2b883a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dadc9384397645eebe8a77c70a2b883a" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 41377 } }
I20240502 02:01:41.272390 30563 raft_consensus.cc:388] T 09c1ed40361246a39b34cabe0fa9ec65 P dadc9384397645eebe8a77c70a2b883a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 02:01:41.272547 30563 raft_consensus.cc:482] T 09c1ed40361246a39b34cabe0fa9ec65 P dadc9384397645eebe8a77c70a2b883a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 02:01:41.272661 30563 raft_consensus.cc:3037] T 09c1ed40361246a39b34cabe0fa9ec65 P dadc9384397645eebe8a77c70a2b883a [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:01:41.274650 30563 raft_consensus.cc:504] T 09c1ed40361246a39b34cabe0fa9ec65 P dadc9384397645eebe8a77c70a2b883a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dadc9384397645eebe8a77c70a2b883a" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 41377 } }
I20240502 02:01:41.274865 30563 leader_election.cc:302] T 09c1ed40361246a39b34cabe0fa9ec65 P dadc9384397645eebe8a77c70a2b883a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dadc9384397645eebe8a77c70a2b883a; no voters: 
I20240502 02:01:41.275036 30563 leader_election.cc:288] T 09c1ed40361246a39b34cabe0fa9ec65 P dadc9384397645eebe8a77c70a2b883a [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 02:01:41.275130 30565 raft_consensus.cc:2781] T 09c1ed40361246a39b34cabe0fa9ec65 P dadc9384397645eebe8a77c70a2b883a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:01:41.275288 30565 raft_consensus.cc:686] T 09c1ed40361246a39b34cabe0fa9ec65 P dadc9384397645eebe8a77c70a2b883a [term 1 LEADER]: Becoming Leader. State: Replica: dadc9384397645eebe8a77c70a2b883a, State: Running, Role: LEADER
I20240502 02:01:41.275490 30563 ts_tablet_manager.cc:1398] T 09c1ed40361246a39b34cabe0fa9ec65 P dadc9384397645eebe8a77c70a2b883a: Time spent starting tablet: real 0.005s	user 0.000s	sys 0.002s
I20240502 02:01:41.275548 30565 consensus_queue.cc:237] T 09c1ed40361246a39b34cabe0fa9ec65 P dadc9384397645eebe8a77c70a2b883a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dadc9384397645eebe8a77c70a2b883a" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 41377 } }
I20240502 02:01:41.277662 29938 catalog_manager.cc:5484] T 09c1ed40361246a39b34cabe0fa9ec65 P dadc9384397645eebe8a77c70a2b883a reported cstate change: term changed from 0 to 1, leader changed from <none> to dadc9384397645eebe8a77c70a2b883a (127.25.109.129). New cstate: current_term: 1 leader_uuid: "dadc9384397645eebe8a77c70a2b883a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dadc9384397645eebe8a77c70a2b883a" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 41377 } } }
I20240502 02:01:41.801291 29938 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43150:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 02:01:41.810674 30195 tablet_service.cc:1467] Processing CreateTablet for tablet ca4b2f70e26243dd9d33ac51a57c7714 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6b20b666e9f94e728cc74308aaefd3e4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 02:01:41.810644 30197 tablet_service.cc:1467] Processing CreateTablet for tablet d890e4aaf028472ab5d44ca2bdd4f388 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6b20b666e9f94e728cc74308aaefd3e4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 02:01:41.810617 30196 tablet_service.cc:1467] Processing CreateTablet for tablet 8d25c2e7da114b2ca6cae050570ad83b (DEFAULT_TABLE table=rebalance_test_table_1 [id=6b20b666e9f94e728cc74308aaefd3e4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 02:01:41.811209 30195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca4b2f70e26243dd9d33ac51a57c7714. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:41.811437 30197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d890e4aaf028472ab5d44ca2bdd4f388. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:41.811692 30196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d25c2e7da114b2ca6cae050570ad83b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:41.817549 30597 tablet_bootstrap.cc:492] T 8d25c2e7da114b2ca6cae050570ad83b P a49cde355f1e4825989af6cb6c874aee: Bootstrap starting.
I20240502 02:01:41.819713 30597 tablet_bootstrap.cc:654] T 8d25c2e7da114b2ca6cae050570ad83b P a49cde355f1e4825989af6cb6c874aee: Neither blocks nor log segments found. Creating new log.
I20240502 02:01:41.821463 30597 tablet_bootstrap.cc:492] T 8d25c2e7da114b2ca6cae050570ad83b P a49cde355f1e4825989af6cb6c874aee: No bootstrap required, opened a new log
I20240502 02:01:41.821614 30597 ts_tablet_manager.cc:1367] T 8d25c2e7da114b2ca6cae050570ad83b P a49cde355f1e4825989af6cb6c874aee: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240502 02:01:41.822562 30597 raft_consensus.cc:348] T 8d25c2e7da114b2ca6cae050570ad83b P a49cde355f1e4825989af6cb6c874aee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49cde355f1e4825989af6cb6c874aee" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 33089 } }
I20240502 02:01:41.822706 30597 raft_consensus.cc:374] T 8d25c2e7da114b2ca6cae050570ad83b P a49cde355f1e4825989af6cb6c874aee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:01:41.822773 30597 raft_consensus.cc:729] T 8d25c2e7da114b2ca6cae050570ad83b P a49cde355f1e4825989af6cb6c874aee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a49cde355f1e4825989af6cb6c874aee, State: Initialized, Role: FOLLOWER
I20240502 02:01:41.822970 30597 consensus_queue.cc:260] T 8d25c2e7da114b2ca6cae050570ad83b P a49cde355f1e4825989af6cb6c874aee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49cde355f1e4825989af6cb6c874aee" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 33089 } }
I20240502 02:01:41.823122 30597 raft_consensus.cc:388] T 8d25c2e7da114b2ca6cae050570ad83b P a49cde355f1e4825989af6cb6c874aee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 02:01:41.823179 30597 raft_consensus.cc:482] T 8d25c2e7da114b2ca6cae050570ad83b P a49cde355f1e4825989af6cb6c874aee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 02:01:41.823256 30597 raft_consensus.cc:3037] T 8d25c2e7da114b2ca6cae050570ad83b P a49cde355f1e4825989af6cb6c874aee [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:01:41.825124 30597 raft_consensus.cc:504] T 8d25c2e7da114b2ca6cae050570ad83b P a49cde355f1e4825989af6cb6c874aee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49cde355f1e4825989af6cb6c874aee" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 33089 } }
I20240502 02:01:41.825394 30597 leader_election.cc:302] T 8d25c2e7da114b2ca6cae050570ad83b P a49cde355f1e4825989af6cb6c874aee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a49cde355f1e4825989af6cb6c874aee; no voters: 
I20240502 02:01:41.825882 30597 leader_election.cc:288] T 8d25c2e7da114b2ca6cae050570ad83b P a49cde355f1e4825989af6cb6c874aee [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 02:01:41.826066 30599 raft_consensus.cc:2781] T 8d25c2e7da114b2ca6cae050570ad83b P a49cde355f1e4825989af6cb6c874aee [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:01:41.826694 30597 ts_tablet_manager.cc:1398] T 8d25c2e7da114b2ca6cae050570ad83b P a49cde355f1e4825989af6cb6c874aee: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.000s
I20240502 02:01:41.826812 30599 raft_consensus.cc:686] T 8d25c2e7da114b2ca6cae050570ad83b P a49cde355f1e4825989af6cb6c874aee [term 1 LEADER]: Becoming Leader. State: Replica: a49cde355f1e4825989af6cb6c874aee, State: Running, Role: LEADER
I20240502 02:01:41.826943 30597 tablet_bootstrap.cc:492] T ca4b2f70e26243dd9d33ac51a57c7714 P a49cde355f1e4825989af6cb6c874aee: Bootstrap starting.
I20240502 02:01:41.827126 30599 consensus_queue.cc:237] T 8d25c2e7da114b2ca6cae050570ad83b P a49cde355f1e4825989af6cb6c874aee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49cde355f1e4825989af6cb6c874aee" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 33089 } }
I20240502 02:01:41.828564 30597 tablet_bootstrap.cc:654] T ca4b2f70e26243dd9d33ac51a57c7714 P a49cde355f1e4825989af6cb6c874aee: Neither blocks nor log segments found. Creating new log.
I20240502 02:01:41.829792 29938 catalog_manager.cc:5484] T 8d25c2e7da114b2ca6cae050570ad83b P a49cde355f1e4825989af6cb6c874aee reported cstate change: term changed from 0 to 1, leader changed from <none> to a49cde355f1e4825989af6cb6c874aee (127.25.109.130). New cstate: current_term: 1 leader_uuid: "a49cde355f1e4825989af6cb6c874aee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49cde355f1e4825989af6cb6c874aee" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 33089 } } }
I20240502 02:01:41.833231 30597 tablet_bootstrap.cc:492] T ca4b2f70e26243dd9d33ac51a57c7714 P a49cde355f1e4825989af6cb6c874aee: No bootstrap required, opened a new log
I20240502 02:01:41.833408 30597 ts_tablet_manager.cc:1367] T ca4b2f70e26243dd9d33ac51a57c7714 P a49cde355f1e4825989af6cb6c874aee: Time spent bootstrapping tablet: real 0.006s	user 0.005s	sys 0.000s
I20240502 02:01:41.834278 30597 raft_consensus.cc:348] T ca4b2f70e26243dd9d33ac51a57c7714 P a49cde355f1e4825989af6cb6c874aee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49cde355f1e4825989af6cb6c874aee" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 33089 } }
I20240502 02:01:41.834465 30597 raft_consensus.cc:374] T ca4b2f70e26243dd9d33ac51a57c7714 P a49cde355f1e4825989af6cb6c874aee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:01:41.834550 30597 raft_consensus.cc:729] T ca4b2f70e26243dd9d33ac51a57c7714 P a49cde355f1e4825989af6cb6c874aee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a49cde355f1e4825989af6cb6c874aee, State: Initialized, Role: FOLLOWER
I20240502 02:01:41.834903 30597 consensus_queue.cc:260] T ca4b2f70e26243dd9d33ac51a57c7714 P a49cde355f1e4825989af6cb6c874aee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49cde355f1e4825989af6cb6c874aee" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 33089 } }
I20240502 02:01:41.835057 30597 raft_consensus.cc:388] T ca4b2f70e26243dd9d33ac51a57c7714 P a49cde355f1e4825989af6cb6c874aee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 02:01:41.835155 30597 raft_consensus.cc:482] T ca4b2f70e26243dd9d33ac51a57c7714 P a49cde355f1e4825989af6cb6c874aee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 02:01:41.835268 30597 raft_consensus.cc:3037] T ca4b2f70e26243dd9d33ac51a57c7714 P a49cde355f1e4825989af6cb6c874aee [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:01:41.837256 30597 raft_consensus.cc:504] T ca4b2f70e26243dd9d33ac51a57c7714 P a49cde355f1e4825989af6cb6c874aee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49cde355f1e4825989af6cb6c874aee" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 33089 } }
I20240502 02:01:41.837453 30597 leader_election.cc:302] T ca4b2f70e26243dd9d33ac51a57c7714 P a49cde355f1e4825989af6cb6c874aee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a49cde355f1e4825989af6cb6c874aee; no voters: 
I20240502 02:01:41.837658 30597 leader_election.cc:288] T ca4b2f70e26243dd9d33ac51a57c7714 P a49cde355f1e4825989af6cb6c874aee [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 02:01:41.837749 30599 raft_consensus.cc:2781] T ca4b2f70e26243dd9d33ac51a57c7714 P a49cde355f1e4825989af6cb6c874aee [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:01:41.838071 30597 ts_tablet_manager.cc:1398] T ca4b2f70e26243dd9d33ac51a57c7714 P a49cde355f1e4825989af6cb6c874aee: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.000s
I20240502 02:01:41.838145 30599 raft_consensus.cc:686] T ca4b2f70e26243dd9d33ac51a57c7714 P a49cde355f1e4825989af6cb6c874aee [term 1 LEADER]: Becoming Leader. State: Replica: a49cde355f1e4825989af6cb6c874aee, State: Running, Role: LEADER
I20240502 02:01:41.838395 30597 tablet_bootstrap.cc:492] T d890e4aaf028472ab5d44ca2bdd4f388 P a49cde355f1e4825989af6cb6c874aee: Bootstrap starting.
I20240502 02:01:41.838390 30599 consensus_queue.cc:237] T ca4b2f70e26243dd9d33ac51a57c7714 P a49cde355f1e4825989af6cb6c874aee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49cde355f1e4825989af6cb6c874aee" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 33089 } }
I20240502 02:01:41.840236 30597 tablet_bootstrap.cc:654] T d890e4aaf028472ab5d44ca2bdd4f388 P a49cde355f1e4825989af6cb6c874aee: Neither blocks nor log segments found. Creating new log.
I20240502 02:01:41.840631 29938 catalog_manager.cc:5484] T ca4b2f70e26243dd9d33ac51a57c7714 P a49cde355f1e4825989af6cb6c874aee reported cstate change: term changed from 0 to 1, leader changed from <none> to a49cde355f1e4825989af6cb6c874aee (127.25.109.130). New cstate: current_term: 1 leader_uuid: "a49cde355f1e4825989af6cb6c874aee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49cde355f1e4825989af6cb6c874aee" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 33089 } } }
I20240502 02:01:41.842307 30597 tablet_bootstrap.cc:492] T d890e4aaf028472ab5d44ca2bdd4f388 P a49cde355f1e4825989af6cb6c874aee: No bootstrap required, opened a new log
I20240502 02:01:41.842447 30597 ts_tablet_manager.cc:1367] T d890e4aaf028472ab5d44ca2bdd4f388 P a49cde355f1e4825989af6cb6c874aee: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240502 02:01:41.843087 30597 raft_consensus.cc:348] T d890e4aaf028472ab5d44ca2bdd4f388 P a49cde355f1e4825989af6cb6c874aee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49cde355f1e4825989af6cb6c874aee" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 33089 } }
I20240502 02:01:41.843222 30597 raft_consensus.cc:374] T d890e4aaf028472ab5d44ca2bdd4f388 P a49cde355f1e4825989af6cb6c874aee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:01:41.843317 30597 raft_consensus.cc:729] T d890e4aaf028472ab5d44ca2bdd4f388 P a49cde355f1e4825989af6cb6c874aee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a49cde355f1e4825989af6cb6c874aee, State: Initialized, Role: FOLLOWER
I20240502 02:01:41.843551 30597 consensus_queue.cc:260] T d890e4aaf028472ab5d44ca2bdd4f388 P a49cde355f1e4825989af6cb6c874aee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49cde355f1e4825989af6cb6c874aee" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 33089 } }
I20240502 02:01:41.843741 30597 raft_consensus.cc:388] T d890e4aaf028472ab5d44ca2bdd4f388 P a49cde355f1e4825989af6cb6c874aee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 02:01:41.843813 30597 raft_consensus.cc:482] T d890e4aaf028472ab5d44ca2bdd4f388 P a49cde355f1e4825989af6cb6c874aee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 02:01:41.843940 30597 raft_consensus.cc:3037] T d890e4aaf028472ab5d44ca2bdd4f388 P a49cde355f1e4825989af6cb6c874aee [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:01:41.845728 30597 raft_consensus.cc:504] T d890e4aaf028472ab5d44ca2bdd4f388 P a49cde355f1e4825989af6cb6c874aee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49cde355f1e4825989af6cb6c874aee" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 33089 } }
I20240502 02:01:41.845949 30597 leader_election.cc:302] T d890e4aaf028472ab5d44ca2bdd4f388 P a49cde355f1e4825989af6cb6c874aee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a49cde355f1e4825989af6cb6c874aee; no voters: 
I20240502 02:01:41.846114 30597 leader_election.cc:288] T d890e4aaf028472ab5d44ca2bdd4f388 P a49cde355f1e4825989af6cb6c874aee [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 02:01:41.846208 30599 raft_consensus.cc:2781] T d890e4aaf028472ab5d44ca2bdd4f388 P a49cde355f1e4825989af6cb6c874aee [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:01:41.846524 30599 raft_consensus.cc:686] T d890e4aaf028472ab5d44ca2bdd4f388 P a49cde355f1e4825989af6cb6c874aee [term 1 LEADER]: Becoming Leader. State: Replica: a49cde355f1e4825989af6cb6c874aee, State: Running, Role: LEADER
I20240502 02:01:41.846524 30597 ts_tablet_manager.cc:1398] T d890e4aaf028472ab5d44ca2bdd4f388 P a49cde355f1e4825989af6cb6c874aee: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.002s
I20240502 02:01:41.846756 30599 consensus_queue.cc:237] T d890e4aaf028472ab5d44ca2bdd4f388 P a49cde355f1e4825989af6cb6c874aee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49cde355f1e4825989af6cb6c874aee" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 33089 } }
I20240502 02:01:41.848791 29937 catalog_manager.cc:5484] T d890e4aaf028472ab5d44ca2bdd4f388 P a49cde355f1e4825989af6cb6c874aee reported cstate change: term changed from 0 to 1, leader changed from <none> to a49cde355f1e4825989af6cb6c874aee (127.25.109.130). New cstate: current_term: 1 leader_uuid: "a49cde355f1e4825989af6cb6c874aee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49cde355f1e4825989af6cb6c874aee" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 33089 } } }
I20240502 02:01:42.378388 29937 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43150:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 02:01:42.388105 30058 tablet_service.cc:1467] Processing CreateTablet for tablet 64a95f86d85b4d6db667d16803527d0d (DEFAULT_TABLE table=rebalance_test_table_2 [id=bb0f231f6cfd4e91b89294ecd9c3091d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 02:01:42.388078 30060 tablet_service.cc:1467] Processing CreateTablet for tablet 920d50c9df77400090829f95f8089c45 (DEFAULT_TABLE table=rebalance_test_table_2 [id=bb0f231f6cfd4e91b89294ecd9c3091d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 02:01:42.388291 30059 tablet_service.cc:1467] Processing CreateTablet for tablet f6f5b01460eb49d6b3173f3312436300 (DEFAULT_TABLE table=rebalance_test_table_2 [id=bb0f231f6cfd4e91b89294ecd9c3091d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 02:01:42.388641 30058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64a95f86d85b4d6db667d16803527d0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:42.388880 30059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6f5b01460eb49d6b3173f3312436300. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:42.389128 30060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 920d50c9df77400090829f95f8089c45. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:42.395964 30633 tablet_bootstrap.cc:492] T 920d50c9df77400090829f95f8089c45 P dadc9384397645eebe8a77c70a2b883a: Bootstrap starting.
I20240502 02:01:42.397397 30633 tablet_bootstrap.cc:654] T 920d50c9df77400090829f95f8089c45 P dadc9384397645eebe8a77c70a2b883a: Neither blocks nor log segments found. Creating new log.
I20240502 02:01:42.399046 30633 tablet_bootstrap.cc:492] T 920d50c9df77400090829f95f8089c45 P dadc9384397645eebe8a77c70a2b883a: No bootstrap required, opened a new log
I20240502 02:01:42.399201 30633 ts_tablet_manager.cc:1367] T 920d50c9df77400090829f95f8089c45 P dadc9384397645eebe8a77c70a2b883a: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 02:01:42.399996 30633 raft_consensus.cc:348] T 920d50c9df77400090829f95f8089c45 P dadc9384397645eebe8a77c70a2b883a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dadc9384397645eebe8a77c70a2b883a" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 41377 } }
I20240502 02:01:42.400167 30633 raft_consensus.cc:374] T 920d50c9df77400090829f95f8089c45 P dadc9384397645eebe8a77c70a2b883a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:01:42.400260 30633 raft_consensus.cc:729] T 920d50c9df77400090829f95f8089c45 P dadc9384397645eebe8a77c70a2b883a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dadc9384397645eebe8a77c70a2b883a, State: Initialized, Role: FOLLOWER
I20240502 02:01:42.400494 30633 consensus_queue.cc:260] T 920d50c9df77400090829f95f8089c45 P dadc9384397645eebe8a77c70a2b883a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dadc9384397645eebe8a77c70a2b883a" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 41377 } }
I20240502 02:01:42.400661 30633 raft_consensus.cc:388] T 920d50c9df77400090829f95f8089c45 P dadc9384397645eebe8a77c70a2b883a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 02:01:42.400732 30633 raft_consensus.cc:482] T 920d50c9df77400090829f95f8089c45 P dadc9384397645eebe8a77c70a2b883a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 02:01:42.400839 30633 raft_consensus.cc:3037] T 920d50c9df77400090829f95f8089c45 P dadc9384397645eebe8a77c70a2b883a [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:01:42.402680 30633 raft_consensus.cc:504] T 920d50c9df77400090829f95f8089c45 P dadc9384397645eebe8a77c70a2b883a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dadc9384397645eebe8a77c70a2b883a" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 41377 } }
I20240502 02:01:42.402921 30633 leader_election.cc:302] T 920d50c9df77400090829f95f8089c45 P dadc9384397645eebe8a77c70a2b883a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dadc9384397645eebe8a77c70a2b883a; no voters: 
I20240502 02:01:42.403357 30633 leader_election.cc:288] T 920d50c9df77400090829f95f8089c45 P dadc9384397645eebe8a77c70a2b883a [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 02:01:42.403527 30635 raft_consensus.cc:2781] T 920d50c9df77400090829f95f8089c45 P dadc9384397645eebe8a77c70a2b883a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:01:42.404175 30633 ts_tablet_manager.cc:1398] T 920d50c9df77400090829f95f8089c45 P dadc9384397645eebe8a77c70a2b883a: Time spent starting tablet: real 0.005s	user 0.001s	sys 0.004s
I20240502 02:01:42.404485 30633 tablet_bootstrap.cc:492] T f6f5b01460eb49d6b3173f3312436300 P dadc9384397645eebe8a77c70a2b883a: Bootstrap starting.
I20240502 02:01:42.404673 30635 raft_consensus.cc:686] T 920d50c9df77400090829f95f8089c45 P dadc9384397645eebe8a77c70a2b883a [term 1 LEADER]: Becoming Leader. State: Replica: dadc9384397645eebe8a77c70a2b883a, State: Running, Role: LEADER
I20240502 02:01:42.404958 30635 consensus_queue.cc:237] T 920d50c9df77400090829f95f8089c45 P dadc9384397645eebe8a77c70a2b883a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dadc9384397645eebe8a77c70a2b883a" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 41377 } }
I20240502 02:01:42.406168 30633 tablet_bootstrap.cc:654] T f6f5b01460eb49d6b3173f3312436300 P dadc9384397645eebe8a77c70a2b883a: Neither blocks nor log segments found. Creating new log.
I20240502 02:01:42.407516 29937 catalog_manager.cc:5484] T 920d50c9df77400090829f95f8089c45 P dadc9384397645eebe8a77c70a2b883a reported cstate change: term changed from 0 to 1, leader changed from <none> to dadc9384397645eebe8a77c70a2b883a (127.25.109.129). New cstate: current_term: 1 leader_uuid: "dadc9384397645eebe8a77c70a2b883a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dadc9384397645eebe8a77c70a2b883a" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 41377 } } }
I20240502 02:01:42.408202 30633 tablet_bootstrap.cc:492] T f6f5b01460eb49d6b3173f3312436300 P dadc9384397645eebe8a77c70a2b883a: No bootstrap required, opened a new log
I20240502 02:01:42.408360 30633 ts_tablet_manager.cc:1367] T f6f5b01460eb49d6b3173f3312436300 P dadc9384397645eebe8a77c70a2b883a: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240502 02:01:42.409147 30633 raft_consensus.cc:348] T f6f5b01460eb49d6b3173f3312436300 P dadc9384397645eebe8a77c70a2b883a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dadc9384397645eebe8a77c70a2b883a" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 41377 } }
I20240502 02:01:42.409330 30633 raft_consensus.cc:374] T f6f5b01460eb49d6b3173f3312436300 P dadc9384397645eebe8a77c70a2b883a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:01:42.409413 30633 raft_consensus.cc:729] T f6f5b01460eb49d6b3173f3312436300 P dadc9384397645eebe8a77c70a2b883a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dadc9384397645eebe8a77c70a2b883a, State: Initialized, Role: FOLLOWER
I20240502 02:01:42.409646 30633 consensus_queue.cc:260] T f6f5b01460eb49d6b3173f3312436300 P dadc9384397645eebe8a77c70a2b883a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dadc9384397645eebe8a77c70a2b883a" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 41377 } }
I20240502 02:01:42.409788 30633 raft_consensus.cc:388] T f6f5b01460eb49d6b3173f3312436300 P dadc9384397645eebe8a77c70a2b883a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 02:01:42.409888 30633 raft_consensus.cc:482] T f6f5b01460eb49d6b3173f3312436300 P dadc9384397645eebe8a77c70a2b883a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 02:01:42.409974 30633 raft_consensus.cc:3037] T f6f5b01460eb49d6b3173f3312436300 P dadc9384397645eebe8a77c70a2b883a [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:01:42.412086 30633 raft_consensus.cc:504] T f6f5b01460eb49d6b3173f3312436300 P dadc9384397645eebe8a77c70a2b883a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dadc9384397645eebe8a77c70a2b883a" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 41377 } }
I20240502 02:01:42.412328 30633 leader_election.cc:302] T f6f5b01460eb49d6b3173f3312436300 P dadc9384397645eebe8a77c70a2b883a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dadc9384397645eebe8a77c70a2b883a; no voters: 
I20240502 02:01:42.412519 30633 leader_election.cc:288] T f6f5b01460eb49d6b3173f3312436300 P dadc9384397645eebe8a77c70a2b883a [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 02:01:42.412617 30635 raft_consensus.cc:2781] T f6f5b01460eb49d6b3173f3312436300 P dadc9384397645eebe8a77c70a2b883a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:01:42.412820 30635 raft_consensus.cc:686] T f6f5b01460eb49d6b3173f3312436300 P dadc9384397645eebe8a77c70a2b883a [term 1 LEADER]: Becoming Leader. State: Replica: dadc9384397645eebe8a77c70a2b883a, State: Running, Role: LEADER
I20240502 02:01:42.413012 30633 ts_tablet_manager.cc:1398] T f6f5b01460eb49d6b3173f3312436300 P dadc9384397645eebe8a77c70a2b883a: Time spent starting tablet: real 0.005s	user 0.001s	sys 0.002s
I20240502 02:01:42.413113 30635 consensus_queue.cc:237] T f6f5b01460eb49d6b3173f3312436300 P dadc9384397645eebe8a77c70a2b883a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dadc9384397645eebe8a77c70a2b883a" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 41377 } }
I20240502 02:01:42.413282 30633 tablet_bootstrap.cc:492] T 64a95f86d85b4d6db667d16803527d0d P dadc9384397645eebe8a77c70a2b883a: Bootstrap starting.
W20240502 02:01:42.413841 30130 tablet.cc:2369] T f6f5b01460eb49d6b3173f3312436300 P dadc9384397645eebe8a77c70a2b883a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 02:01:42.415055 30633 tablet_bootstrap.cc:654] T 64a95f86d85b4d6db667d16803527d0d P dadc9384397645eebe8a77c70a2b883a: Neither blocks nor log segments found. Creating new log.
I20240502 02:01:42.416150 29937 catalog_manager.cc:5484] T f6f5b01460eb49d6b3173f3312436300 P dadc9384397645eebe8a77c70a2b883a reported cstate change: term changed from 0 to 1, leader changed from <none> to dadc9384397645eebe8a77c70a2b883a (127.25.109.129). New cstate: current_term: 1 leader_uuid: "dadc9384397645eebe8a77c70a2b883a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dadc9384397645eebe8a77c70a2b883a" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 41377 } } }
I20240502 02:01:42.417073 30633 tablet_bootstrap.cc:492] T 64a95f86d85b4d6db667d16803527d0d P dadc9384397645eebe8a77c70a2b883a: No bootstrap required, opened a new log
I20240502 02:01:42.417222 30633 ts_tablet_manager.cc:1367] T 64a95f86d85b4d6db667d16803527d0d P dadc9384397645eebe8a77c70a2b883a: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.003s
I20240502 02:01:42.417943 30633 raft_consensus.cc:348] T 64a95f86d85b4d6db667d16803527d0d P dadc9384397645eebe8a77c70a2b883a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dadc9384397645eebe8a77c70a2b883a" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 41377 } }
I20240502 02:01:42.418087 30633 raft_consensus.cc:374] T 64a95f86d85b4d6db667d16803527d0d P dadc9384397645eebe8a77c70a2b883a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:01:42.418177 30633 raft_consensus.cc:729] T 64a95f86d85b4d6db667d16803527d0d P dadc9384397645eebe8a77c70a2b883a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dadc9384397645eebe8a77c70a2b883a, State: Initialized, Role: FOLLOWER
I20240502 02:01:42.418409 30633 consensus_queue.cc:260] T 64a95f86d85b4d6db667d16803527d0d P dadc9384397645eebe8a77c70a2b883a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dadc9384397645eebe8a77c70a2b883a" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 41377 } }
I20240502 02:01:42.418617 30633 raft_consensus.cc:388] T 64a95f86d85b4d6db667d16803527d0d P dadc9384397645eebe8a77c70a2b883a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 02:01:42.418709 30633 raft_consensus.cc:482] T 64a95f86d85b4d6db667d16803527d0d P dadc9384397645eebe8a77c70a2b883a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 02:01:42.418862 30633 raft_consensus.cc:3037] T 64a95f86d85b4d6db667d16803527d0d P dadc9384397645eebe8a77c70a2b883a [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:01:42.420843 30633 raft_consensus.cc:504] T 64a95f86d85b4d6db667d16803527d0d P dadc9384397645eebe8a77c70a2b883a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dadc9384397645eebe8a77c70a2b883a" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 41377 } }
I20240502 02:01:42.421079 30633 leader_election.cc:302] T 64a95f86d85b4d6db667d16803527d0d P dadc9384397645eebe8a77c70a2b883a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dadc9384397645eebe8a77c70a2b883a; no voters: 
I20240502 02:01:42.421242 30633 leader_election.cc:288] T 64a95f86d85b4d6db667d16803527d0d P dadc9384397645eebe8a77c70a2b883a [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 02:01:42.421334 30635 raft_consensus.cc:2781] T 64a95f86d85b4d6db667d16803527d0d P dadc9384397645eebe8a77c70a2b883a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:01:42.421500 30635 raft_consensus.cc:686] T 64a95f86d85b4d6db667d16803527d0d P dadc9384397645eebe8a77c70a2b883a [term 1 LEADER]: Becoming Leader. State: Replica: dadc9384397645eebe8a77c70a2b883a, State: Running, Role: LEADER
I20240502 02:01:42.421682 30633 ts_tablet_manager.cc:1398] T 64a95f86d85b4d6db667d16803527d0d P dadc9384397645eebe8a77c70a2b883a: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.003s
I20240502 02:01:42.421747 30635 consensus_queue.cc:237] T 64a95f86d85b4d6db667d16803527d0d P dadc9384397645eebe8a77c70a2b883a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dadc9384397645eebe8a77c70a2b883a" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 41377 } }
I20240502 02:01:42.423816 29938 catalog_manager.cc:5484] T 64a95f86d85b4d6db667d16803527d0d P dadc9384397645eebe8a77c70a2b883a reported cstate change: term changed from 0 to 1, leader changed from <none> to dadc9384397645eebe8a77c70a2b883a (127.25.109.129). New cstate: current_term: 1 leader_uuid: "dadc9384397645eebe8a77c70a2b883a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dadc9384397645eebe8a77c70a2b883a" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 41377 } } }
I20240502 02:01:42.938473 29938 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43150:
name: "rebalance_test_table_3"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 02:01:42.946826 30195 tablet_service.cc:1467] Processing CreateTablet for tablet 2b2efa19565546eab2e79821174c9d20 (DEFAULT_TABLE table=rebalance_test_table_3 [id=eac656f731ec4e068d7afed6a20e5037]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 02:01:42.946826 30197 tablet_service.cc:1467] Processing CreateTablet for tablet e5d4c91515284747b432b06d633b9715 (DEFAULT_TABLE table=rebalance_test_table_3 [id=eac656f731ec4e068d7afed6a20e5037]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 02:01:42.947182 30196 tablet_service.cc:1467] Processing CreateTablet for tablet a25a2d00142a42b69cde36ab4eddf659 (DEFAULT_TABLE table=rebalance_test_table_3 [id=eac656f731ec4e068d7afed6a20e5037]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 02:01:42.947368 30195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b2efa19565546eab2e79821174c9d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:42.947551 30196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a25a2d00142a42b69cde36ab4eddf659. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:42.947813 30197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5d4c91515284747b432b06d633b9715. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:42.953658 30668 tablet_bootstrap.cc:492] T a25a2d00142a42b69cde36ab4eddf659 P a49cde355f1e4825989af6cb6c874aee: Bootstrap starting.
I20240502 02:01:42.955200 30668 tablet_bootstrap.cc:654] T a25a2d00142a42b69cde36ab4eddf659 P a49cde355f1e4825989af6cb6c874aee: Neither blocks nor log segments found. Creating new log.
I20240502 02:01:42.957038 30668 tablet_bootstrap.cc:492] T a25a2d00142a42b69cde36ab4eddf659 P a49cde355f1e4825989af6cb6c874aee: No bootstrap required, opened a new log
I20240502 02:01:42.957187 30668 ts_tablet_manager.cc:1367] T a25a2d00142a42b69cde36ab4eddf659 P a49cde355f1e4825989af6cb6c874aee: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240502 02:01:42.957825 30668 raft_consensus.cc:348] T a25a2d00142a42b69cde36ab4eddf659 P a49cde355f1e4825989af6cb6c874aee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49cde355f1e4825989af6cb6c874aee" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 33089 } }
I20240502 02:01:42.957986 30668 raft_consensus.cc:374] T a25a2d00142a42b69cde36ab4eddf659 P a49cde355f1e4825989af6cb6c874aee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:01:42.958076 30668 raft_consensus.cc:729] T a25a2d00142a42b69cde36ab4eddf659 P a49cde355f1e4825989af6cb6c874aee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a49cde355f1e4825989af6cb6c874aee, State: Initialized, Role: FOLLOWER
I20240502 02:01:42.958320 30668 consensus_queue.cc:260] T a25a2d00142a42b69cde36ab4eddf659 P a49cde355f1e4825989af6cb6c874aee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49cde355f1e4825989af6cb6c874aee" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 33089 } }
I20240502 02:01:42.958485 30668 raft_consensus.cc:388] T a25a2d00142a42b69cde36ab4eddf659 P a49cde355f1e4825989af6cb6c874aee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 02:01:42.958570 30668 raft_consensus.cc:482] T a25a2d00142a42b69cde36ab4eddf659 P a49cde355f1e4825989af6cb6c874aee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 02:01:42.958676 30668 raft_consensus.cc:3037] T a25a2d00142a42b69cde36ab4eddf659 P a49cde355f1e4825989af6cb6c874aee [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:01:42.960934 30668 raft_consensus.cc:504] T a25a2d00142a42b69cde36ab4eddf659 P a49cde355f1e4825989af6cb6c874aee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49cde355f1e4825989af6cb6c874aee" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 33089 } }
I20240502 02:01:42.961182 30668 leader_election.cc:302] T a25a2d00142a42b69cde36ab4eddf659 P a49cde355f1e4825989af6cb6c874aee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a49cde355f1e4825989af6cb6c874aee; no voters: 
I20240502 02:01:42.961604 30668 leader_election.cc:288] T a25a2d00142a42b69cde36ab4eddf659 P a49cde355f1e4825989af6cb6c874aee [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 02:01:42.961802 30670 raft_consensus.cc:2781] T a25a2d00142a42b69cde36ab4eddf659 P a49cde355f1e4825989af6cb6c874aee [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:01:42.962270 30668 ts_tablet_manager.cc:1398] T a25a2d00142a42b69cde36ab4eddf659 P a49cde355f1e4825989af6cb6c874aee: Time spent starting tablet: real 0.005s	user 0.001s	sys 0.004s
I20240502 02:01:42.962466 30670 raft_consensus.cc:686] T a25a2d00142a42b69cde36ab4eddf659 P a49cde355f1e4825989af6cb6c874aee [term 1 LEADER]: Becoming Leader. State: Replica: a49cde355f1e4825989af6cb6c874aee, State: Running, Role: LEADER
I20240502 02:01:42.962579 30668 tablet_bootstrap.cc:492] T 2b2efa19565546eab2e79821174c9d20 P a49cde355f1e4825989af6cb6c874aee: Bootstrap starting.
I20240502 02:01:42.962749 30670 consensus_queue.cc:237] T a25a2d00142a42b69cde36ab4eddf659 P a49cde355f1e4825989af6cb6c874aee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49cde355f1e4825989af6cb6c874aee" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 33089 } }
I20240502 02:01:42.964215 30668 tablet_bootstrap.cc:654] T 2b2efa19565546eab2e79821174c9d20 P a49cde355f1e4825989af6cb6c874aee: Neither blocks nor log segments found. Creating new log.
I20240502 02:01:42.965802 29937 catalog_manager.cc:5484] T a25a2d00142a42b69cde36ab4eddf659 P a49cde355f1e4825989af6cb6c874aee reported cstate change: term changed from 0 to 1, leader changed from <none> to a49cde355f1e4825989af6cb6c874aee (127.25.109.130). New cstate: current_term: 1 leader_uuid: "a49cde355f1e4825989af6cb6c874aee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49cde355f1e4825989af6cb6c874aee" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 33089 } } }
I20240502 02:01:42.967078 30668 tablet_bootstrap.cc:492] T 2b2efa19565546eab2e79821174c9d20 P a49cde355f1e4825989af6cb6c874aee: No bootstrap required, opened a new log
I20240502 02:01:42.967231 30668 ts_tablet_manager.cc:1367] T 2b2efa19565546eab2e79821174c9d20 P a49cde355f1e4825989af6cb6c874aee: Time spent bootstrapping tablet: real 0.005s	user 0.000s	sys 0.004s
I20240502 02:01:42.968040 30668 raft_consensus.cc:348] T 2b2efa19565546eab2e79821174c9d20 P a49cde355f1e4825989af6cb6c874aee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49cde355f1e4825989af6cb6c874aee" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 33089 } }
I20240502 02:01:42.968250 30668 raft_consensus.cc:374] T 2b2efa19565546eab2e79821174c9d20 P a49cde355f1e4825989af6cb6c874aee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:01:42.968338 30668 raft_consensus.cc:729] T 2b2efa19565546eab2e79821174c9d20 P a49cde355f1e4825989af6cb6c874aee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a49cde355f1e4825989af6cb6c874aee, State: Initialized, Role: FOLLOWER
I20240502 02:01:42.968617 30668 consensus_queue.cc:260] T 2b2efa19565546eab2e79821174c9d20 P a49cde355f1e4825989af6cb6c874aee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49cde355f1e4825989af6cb6c874aee" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 33089 } }
I20240502 02:01:42.968771 30668 raft_consensus.cc:388] T 2b2efa19565546eab2e79821174c9d20 P a49cde355f1e4825989af6cb6c874aee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 02:01:42.968868 30668 raft_consensus.cc:482] T 2b2efa19565546eab2e79821174c9d20 P a49cde355f1e4825989af6cb6c874aee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 02:01:42.968963 30668 raft_consensus.cc:3037] T 2b2efa19565546eab2e79821174c9d20 P a49cde355f1e4825989af6cb6c874aee [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:01:42.971230 30668 raft_consensus.cc:504] T 2b2efa19565546eab2e79821174c9d20 P a49cde355f1e4825989af6cb6c874aee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49cde355f1e4825989af6cb6c874aee" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 33089 } }
I20240502 02:01:42.971495 30668 leader_election.cc:302] T 2b2efa19565546eab2e79821174c9d20 P a49cde355f1e4825989af6cb6c874aee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a49cde355f1e4825989af6cb6c874aee; no voters: 
I20240502 02:01:42.971710 30668 leader_election.cc:288] T 2b2efa19565546eab2e79821174c9d20 P a49cde355f1e4825989af6cb6c874aee [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 02:01:42.971771 30670 raft_consensus.cc:2781] T 2b2efa19565546eab2e79821174c9d20 P a49cde355f1e4825989af6cb6c874aee [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:01:42.971983 30670 raft_consensus.cc:686] T 2b2efa19565546eab2e79821174c9d20 P a49cde355f1e4825989af6cb6c874aee [term 1 LEADER]: Becoming Leader. State: Replica: a49cde355f1e4825989af6cb6c874aee, State: Running, Role: LEADER
I20240502 02:01:42.972236 30668 ts_tablet_manager.cc:1398] T 2b2efa19565546eab2e79821174c9d20 P a49cde355f1e4825989af6cb6c874aee: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.000s
I20240502 02:01:42.972244 30670 consensus_queue.cc:237] T 2b2efa19565546eab2e79821174c9d20 P a49cde355f1e4825989af6cb6c874aee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49cde355f1e4825989af6cb6c874aee" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 33089 } }
I20240502 02:01:42.972515 30668 tablet_bootstrap.cc:492] T e5d4c91515284747b432b06d633b9715 P a49cde355f1e4825989af6cb6c874aee: Bootstrap starting.
I20240502 02:01:42.974246 30668 tablet_bootstrap.cc:654] T e5d4c91515284747b432b06d633b9715 P a49cde355f1e4825989af6cb6c874aee: Neither blocks nor log segments found. Creating new log.
I20240502 02:01:42.974682 29938 catalog_manager.cc:5484] T 2b2efa19565546eab2e79821174c9d20 P a49cde355f1e4825989af6cb6c874aee reported cstate change: term changed from 0 to 1, leader changed from <none> to a49cde355f1e4825989af6cb6c874aee (127.25.109.130). New cstate: current_term: 1 leader_uuid: "a49cde355f1e4825989af6cb6c874aee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49cde355f1e4825989af6cb6c874aee" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 33089 } } }
I20240502 02:01:42.976085 30668 tablet_bootstrap.cc:492] T e5d4c91515284747b432b06d633b9715 P a49cde355f1e4825989af6cb6c874aee: No bootstrap required, opened a new log
I20240502 02:01:42.976213 30668 ts_tablet_manager.cc:1367] T e5d4c91515284747b432b06d633b9715 P a49cde355f1e4825989af6cb6c874aee: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240502 02:01:42.976899 30668 raft_consensus.cc:348] T e5d4c91515284747b432b06d633b9715 P a49cde355f1e4825989af6cb6c874aee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49cde355f1e4825989af6cb6c874aee" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 33089 } }
I20240502 02:01:42.977088 30668 raft_consensus.cc:374] T e5d4c91515284747b432b06d633b9715 P a49cde355f1e4825989af6cb6c874aee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:01:42.977183 30668 raft_consensus.cc:729] T e5d4c91515284747b432b06d633b9715 P a49cde355f1e4825989af6cb6c874aee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a49cde355f1e4825989af6cb6c874aee, State: Initialized, Role: FOLLOWER
I20240502 02:01:42.977407 30668 consensus_queue.cc:260] T e5d4c91515284747b432b06d633b9715 P a49cde355f1e4825989af6cb6c874aee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49cde355f1e4825989af6cb6c874aee" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 33089 } }
I20240502 02:01:42.977630 30668 raft_consensus.cc:388] T e5d4c91515284747b432b06d633b9715 P a49cde355f1e4825989af6cb6c874aee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 02:01:42.977713 30668 raft_consensus.cc:482] T e5d4c91515284747b432b06d633b9715 P a49cde355f1e4825989af6cb6c874aee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 02:01:42.977813 30668 raft_consensus.cc:3037] T e5d4c91515284747b432b06d633b9715 P a49cde355f1e4825989af6cb6c874aee [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:01:42.980264 30668 raft_consensus.cc:504] T e5d4c91515284747b432b06d633b9715 P a49cde355f1e4825989af6cb6c874aee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49cde355f1e4825989af6cb6c874aee" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 33089 } }
I20240502 02:01:42.980576 30668 leader_election.cc:302] T e5d4c91515284747b432b06d633b9715 P a49cde355f1e4825989af6cb6c874aee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a49cde355f1e4825989af6cb6c874aee; no voters: 
I20240502 02:01:42.980767 30668 leader_election.cc:288] T e5d4c91515284747b432b06d633b9715 P a49cde355f1e4825989af6cb6c874aee [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 02:01:42.980855 30670 raft_consensus.cc:2781] T e5d4c91515284747b432b06d633b9715 P a49cde355f1e4825989af6cb6c874aee [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:01:42.981007 30670 raft_consensus.cc:686] T e5d4c91515284747b432b06d633b9715 P a49cde355f1e4825989af6cb6c874aee [term 1 LEADER]: Becoming Leader. State: Replica: a49cde355f1e4825989af6cb6c874aee, State: Running, Role: LEADER
I20240502 02:01:42.981315 30668 ts_tablet_manager.cc:1398] T e5d4c91515284747b432b06d633b9715 P a49cde355f1e4825989af6cb6c874aee: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20240502 02:01:42.981261 30670 consensus_queue.cc:237] T e5d4c91515284747b432b06d633b9715 P a49cde355f1e4825989af6cb6c874aee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49cde355f1e4825989af6cb6c874aee" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 33089 } }
I20240502 02:01:42.983465 29938 catalog_manager.cc:5484] T e5d4c91515284747b432b06d633b9715 P a49cde355f1e4825989af6cb6c874aee reported cstate change: term changed from 0 to 1, leader changed from <none> to a49cde355f1e4825989af6cb6c874aee (127.25.109.130). New cstate: current_term: 1 leader_uuid: "a49cde355f1e4825989af6cb6c874aee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a49cde355f1e4825989af6cb6c874aee" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 33089 } } }
I20240502 02:01:43.511950 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.131:38831
--local_ip_for_outbound_sockets=127.25.109.131
--tserver_master_addrs=127.25.109.190:34625
--webserver_port=33085
--webserver_interface=127.25.109.131
--builtin_ntp_servers=127.25.109.148:36075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 02:01:43.862390 30701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:01:43.862744 30701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 02:01:43.862883 30701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:01:43.870888 30701 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 02:01:43.871049 30701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 02:01:43.871210 30701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.131
I20240502 02:01:43.880848 30701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:36075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.131:38831
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.25.109.131
--webserver_port=33085
--tserver_master_addrs=127.25.109.190:34625
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.131
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 02:01:43.882248 30701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 02:01:43.884100 30701 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 02:01:43.896245 30706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:01:43.896641 30707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:01:43.897312 30709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 02:01:43.897842 30701 server_base.cc:1034] running on GCE node
I20240502 02:01:43.898478 30701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 02:01:43.899469 30701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 02:01:43.900692 30701 hybrid_clock.cc:648] HybridClock initialized: now 1714615303900628 us; error 61 us; skew 500 ppm
I20240502 02:01:43.901039 30701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 02:01:43.903399 30701 webserver.cc:460] Webserver started at http://127.25.109.131:33085/ using document root <none> and password file <none>
I20240502 02:01:43.904246 30701 fs_manager.cc:359] Metadata directory not provided
I20240502 02:01:43.904393 30701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 02:01:43.909868 30701 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240502 02:01:43.913017 30715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:01:43.914520 30701 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 02:01:43.914721 30701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "59daeed1fd4d4db5b7b41e678bb22c21"
format_stamp: "Formatted at 2024-05-02 02:01:36 on dist-test-slave-kj4b"
I20240502 02:01:43.915616 30701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 02:01:43.945925 30701 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 02:01:43.946817 30701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 02:01:43.947126 30701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 02:01:43.948125 30701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 02:01:43.949680 30701 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 02:01:43.949760 30701 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:01:43.949887 30701 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 02:01:43.949939 30701 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:01:43.980351 30701 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.131:38831
I20240502 02:01:43.980381 30830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.131:38831 every 8 connection(s)
I20240502 02:01:43.981894 30701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 02:01:43.988243 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 30701
I20240502 02:01:43.989126 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.132:45285
--local_ip_for_outbound_sockets=127.25.109.132
--tserver_master_addrs=127.25.109.190:34625
--webserver_port=39215
--webserver_interface=127.25.109.132
--builtin_ntp_servers=127.25.109.148:36075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 02:01:43.996304 30831 heartbeater.cc:347] Connected to a master server at 127.25.109.190:34625
I20240502 02:01:43.996699 30831 heartbeater.cc:464] Registering TS with master...
I20240502 02:01:43.997658 30831 heartbeater.cc:511] Master 127.25.109.190:34625 requested a full tablet report, sending...
I20240502 02:01:43.999459 29938 ts_manager.cc:194] Re-registered known tserver with Master: 59daeed1fd4d4db5b7b41e678bb22c21 (127.25.109.131:38831)
I20240502 02:01:44.000314 29938 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.131:46571
W20240502 02:01:44.341392 30835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:01:44.341676 30835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 02:01:44.341778 30835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:01:44.349253 30835 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 02:01:44.349393 30835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 02:01:44.349515 30835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.132
I20240502 02:01:44.359102 30835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:36075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.132:45285
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.25.109.132
--webserver_port=39215
--tserver_master_addrs=127.25.109.190:34625
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.132
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 02:01:44.360495 30835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 02:01:44.362291 30835 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 02:01:44.375105 30840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:01:44.375669 30844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:01:44.375696 30841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 02:01:44.376128 30835 server_base.cc:1034] running on GCE node
I20240502 02:01:44.376788 30835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 02:01:44.377833 30835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 02:01:44.379055 30835 hybrid_clock.cc:648] HybridClock initialized: now 1714615304379005 us; error 42 us; skew 500 ppm
I20240502 02:01:44.379405 30835 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 02:01:44.381973 30835 webserver.cc:460] Webserver started at http://127.25.109.132:39215/ using document root <none> and password file <none>
I20240502 02:01:44.382809 30835 fs_manager.cc:359] Metadata directory not provided
I20240502 02:01:44.382972 30835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 02:01:44.388728 30835 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.000s
I20240502 02:01:44.391795 30850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:01:44.393276 30835 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
I20240502 02:01:44.393460 30835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "4180f0eefa0642ad9ab53b9dd07b8cf8"
format_stamp: "Formatted at 2024-05-02 02:01:37 on dist-test-slave-kj4b"
I20240502 02:01:44.394460 30835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 02:01:44.419606 30835 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 02:01:44.420712 30835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 02:01:44.421029 30835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 02:01:44.422086 30835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 02:01:44.423615 30835 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 02:01:44.423692 30835 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:01:44.423790 30835 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 02:01:44.423833 30835 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:01:44.454962 30835 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.132:45285
I20240502 02:01:44.454993 30964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.132:45285 every 8 connection(s)
I20240502 02:01:44.456575 30835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 02:01:44.457755 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 30835
I20240502 02:01:44.470800 30965 heartbeater.cc:347] Connected to a master server at 127.25.109.190:34625
I20240502 02:01:44.471150 30965 heartbeater.cc:464] Registering TS with master...
I20240502 02:01:44.472002 30965 heartbeater.cc:511] Master 127.25.109.190:34625 requested a full tablet report, sending...
I20240502 02:01:44.473524 29938 ts_manager.cc:194] Re-registered known tserver with Master: 4180f0eefa0642ad9ab53b9dd07b8cf8 (127.25.109.132:45285)
I20240502 02:01:44.474393 29938 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.132:58695
I20240502 02:01:44.955989 30765 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:44.957791 30197 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:44.962755 30899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:44.984515 30060 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:45.003545 30831 heartbeater.cc:503] Master 127.25.109.190:34625 was elected leader, sending a full tablet report...
I20240502 02:01:45.159847 30765 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:45.160436 30899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:45.166576 30060 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:45.171516 30197 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:45.345453 30899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:45.352368 30197 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:45.360662 30765 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:45.363690 30060 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:01:45.477234 30965 heartbeater.cc:503] Master 127.25.109.190:34625 was elected leader, sending a full tablet report...
I20240502 02:01:45.619904 30899 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:45.628229 30197 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 02:01:45.630532 30765 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:45.639546 30060 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
Master Summary
               UUID               |       Address        | Status
----------------------------------+----------------------+---------
 cef66a04ddc4490fabb3151c4994b050 | 127.25.109.190:34625 | HEALTHY

Unusual flags for Master:
                   Flag                   |                                                                                               Value                                                                                                |         Tags          |         Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_ca_key_size                         | 768                                                                                                                                                                                                | experimental          | all 1 server(s) checked
 ipki_server_key_size                     | 768                                                                                                                                                                                                | experimental          | all 1 server(s) checked
 never_fsync                              | true                                                                                                                                                                                               | unsafe,advanced       | all 1 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                                                  | unsafe,hidden         | all 1 server(s) checked
 raft_prepare_replacement_before_eviction | false                                                                                                                                                                                              | experimental,advanced | all 1 server(s) checked
 rpc_reuseport                            | true                                                                                                                                                                                               | experimental          | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                                               | unsafe                | all 1 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                                                 | hidden                | all 1 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden                | all 1 server(s) checked
 tsk_num_rsa_bits                         | 512                                                                                                                                                                                                | experimental          | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |        Value         |         Master
---------------------+----------------------+-------------------------
 builtin_ntp_servers | 127.25.109.148:36075 | all 1 server(s) checked
 time_source         | builtin              | all 1 server(s) checked

Tablet Server Summary
               UUID               |       Address        | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
 4180f0eefa0642ad9ab53b9dd07b8cf8 | 127.25.109.132:45285 | HEALTHY | <none>   |       0        |       0
 59daeed1fd4d4db5b7b41e678bb22c21 | 127.25.109.131:38831 | HEALTHY | <none>   |       0        |       0
 a49cde355f1e4825989af6cb6c874aee | 127.25.109.130:33089 | HEALTHY | <none>   |       7        |       0
 dadc9384397645eebe8a77c70a2b883a | 127.25.109.129:41377 | HEALTHY | <none>   |       6        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       4

Unusual flags for Tablet Server:
                   Flag                   |                                                                                             Value                                                                                              |         Tags          |      Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_server_key_size                     | 768                                                                                                                                                                                            | experimental          | all 4 server(s) checked
 local_ip_for_outbound_sockets            | 127.25.109.129                                                                                                                                                                                 | experimental          | 127.25.109.129:41377
 local_ip_for_outbound_sockets            | 127.25.109.130                                                                                                                                                                                 | experimental          | 127.25.109.130:33089
 local_ip_for_outbound_sockets            | 127.25.109.131                                                                                                                                                                                 | experimental          | 127.25.109.131:38831
 local_ip_for_outbound_sockets            | 127.25.109.132                                                                                                                                                                                 | experimental          | 127.25.109.132:45285
 never_fsync                              | true                                                                                                                                                                                           | unsafe,advanced       | all 4 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                                              | unsafe,hidden         | all 4 server(s) checked
 raft_prepare_replacement_before_eviction | false                                                                                                                                                                                          | experimental,advanced | all 4 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                                           | unsafe                | all 4 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                                             | hidden                | all 4 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden                | 127.25.109.129:41377
 server_dump_info_path                    | /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden                | 127.25.109.130:33089
 server_dump_info_path                    | /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden                | 127.25.109.131:38831
 server_dump_info_path                    | /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden                | 127.25.109.132:45285

Flags of checked categories for Tablet Server:
        Flag         |        Value         |      Tablet Server
---------------------+----------------------+-------------------------
 builtin_ntp_servers | 127.25.109.148:36075 | all 4 server(s) checked
 time_source         | builtin              | all 4 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.18.0-SNAPSHOT | all 5 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
          Name          | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
 TestTable              | 1  | HEALTHY | 1             | 1       | 0          | 0                | 0
 rebalance_test_table_0 | 1  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_1 | 1  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_2 | 1  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_3 | 1  | HEALTHY | 3             | 3       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 0
 First Quartile | 0
 Median         | 6
 Third Quartile | 7
 Maximum        | 7

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 4
 Tables         | 5
 Tablets        | 13
 Replicas       | 13

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20240502 02:01:45.689543 26038 log_verifier.cc:126] Checking tablet 09c1ed40361246a39b34cabe0fa9ec65
I20240502 02:01:45.697664 26038 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 09c1ed40361246a39b34cabe0fa9ec65
I20240502 02:01:45.697769 26038 log_verifier.cc:126] Checking tablet 2b2efa19565546eab2e79821174c9d20
I20240502 02:01:45.705389 26038 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 2b2efa19565546eab2e79821174c9d20
I20240502 02:01:45.705490 26038 log_verifier.cc:126] Checking tablet 64a95f86d85b4d6db667d16803527d0d
I20240502 02:01:45.713256 26038 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 64a95f86d85b4d6db667d16803527d0d
I20240502 02:01:45.713351 26038 log_verifier.cc:126] Checking tablet 89a27f63639244129d249d42ae4bb291
I20240502 02:01:45.721099 26038 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 89a27f63639244129d249d42ae4bb291
I20240502 02:01:45.721194 26038 log_verifier.cc:126] Checking tablet 8d25c2e7da114b2ca6cae050570ad83b
I20240502 02:01:45.728930 26038 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 8d25c2e7da114b2ca6cae050570ad83b
I20240502 02:01:45.729024 26038 log_verifier.cc:126] Checking tablet 8fcc05fb4cd949ae96776fe4bfb6317b
I20240502 02:01:45.737128 26038 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 8fcc05fb4cd949ae96776fe4bfb6317b
I20240502 02:01:45.737222 26038 log_verifier.cc:126] Checking tablet 920d50c9df77400090829f95f8089c45
I20240502 02:01:45.745031 26038 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 920d50c9df77400090829f95f8089c45
I20240502 02:01:45.745126 26038 log_verifier.cc:126] Checking tablet a25a2d00142a42b69cde36ab4eddf659
I20240502 02:01:45.752807 26038 log_verifier.cc:177] Verified matching terms for 1 ops in tablet a25a2d00142a42b69cde36ab4eddf659
I20240502 02:01:45.752910 26038 log_verifier.cc:126] Checking tablet ca4b2f70e26243dd9d33ac51a57c7714
I20240502 02:01:45.760704 26038 log_verifier.cc:177] Verified matching terms for 1 ops in tablet ca4b2f70e26243dd9d33ac51a57c7714
I20240502 02:01:45.760804 26038 log_verifier.cc:126] Checking tablet d890e4aaf028472ab5d44ca2bdd4f388
I20240502 02:01:45.769027 26038 log_verifier.cc:177] Verified matching terms for 2 ops in tablet d890e4aaf028472ab5d44ca2bdd4f388
I20240502 02:01:45.769129 26038 log_verifier.cc:126] Checking tablet e5d4c91515284747b432b06d633b9715
I20240502 02:01:45.777351 26038 log_verifier.cc:177] Verified matching terms for 2 ops in tablet e5d4c91515284747b432b06d633b9715
I20240502 02:01:45.777454 26038 log_verifier.cc:126] Checking tablet eebfabb6d8b04aabb2c34cfbb26fe0b9
I20240502 02:01:45.785387 26038 log_verifier.cc:177] Verified matching terms for 1 ops in tablet eebfabb6d8b04aabb2c34cfbb26fe0b9
I20240502 02:01:45.785496 26038 log_verifier.cc:126] Checking tablet f6f5b01460eb49d6b3173f3312436300
I20240502 02:01:45.793416 26038 log_verifier.cc:177] Verified matching terms for 1 ops in tablet f6f5b01460eb49d6b3173f3312436300
I20240502 02:01:45.794370 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 29998
I20240502 02:01:45.795805 30124 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 02:01:45.898952 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 29998
I20240502 02:01:45.917266 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 30133
I20240502 02:01:45.918393 30258 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 02:01:46.027933 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 30133
W20240502 02:01:46.046633 29922 connection.cc:535] server connection from 127.25.109.130:37061 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 02:01:46.046869 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 30701
I20240502 02:01:46.053112 30824 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 02:01:46.143150 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 30701
I20240502 02:01:46.159440 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 30835
I20240502 02:01:46.165664 30960 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 02:01:46.257714 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 30835
I20240502 02:01:46.274257 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 29905
I20240502 02:01:46.275424 29968 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 02:01:46.397909 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 29905
2024-05-02T02:01:46Z chronyd exiting
[       OK ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/0 (11966 ms)
[----------] 1 test from RebalancerAndSingleReplicaTablets (11966 ms total)

[----------] 2 tests from LocationAwareRebalancingParamTest
[ RUN      ] LocationAwareRebalancingParamTest.Rebalance/idx2_rf3_t3_l_3_2
I20240502 02:01:46.498165 26038 test_util.cc:268] Using random seed: -62574489
I20240502 02:01:46.499953 26038 ts_itest-base.cc:112] Starting cluster with:
I20240502 02:01:46.500028 26038 ts_itest-base.cc:113] --------------
I20240502 02:01:46.500059 26038 ts_itest-base.cc:114] 5 tablet servers
I20240502 02:01:46.500082 26038 ts_itest-base.cc:115] 3 replicas per TS
I20240502 02:01:46.500129 26038 ts_itest-base.cc:116] --------------
2024-05-02T02:01:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T02:01:46Z Disabled control of system clock
I20240502 02:01:46.561349 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.190:43455
--webserver_interface=127.25.109.190
--webserver_port=0
--builtin_ntp_servers=127.25.109.148:42757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.190:43455
--location_mapping_cmd=/tmp/dist-test-taskb4pM1q/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:2
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240502 02:01:46.915640 31119 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 02:01:46.916098 31119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:01:46.916204 31119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 02:01:46.916312 31119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:01:46.924122 31119 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 02:01:46.924273 31119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 02:01:46.924372 31119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 02:01:46.924429 31119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 02:01:46.924515 31119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 02:01:46.934070 31119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:42757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskb4pM1q/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:2
--ipki_ca_key_size=768
--master_addresses=127.25.109.190:43455
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.190:43455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.25.109.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 02:01:46.935472 31119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 02:01:46.937296 31119 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 02:01:46.947842 31124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:01:46.948367 31125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:01:46.948699 31127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 02:01:46.948886 31119 server_base.cc:1034] running on GCE node
I20240502 02:01:46.949715 31119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 02:01:46.951224 31119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 02:01:46.952455 31119 hybrid_clock.cc:648] HybridClock initialized: now 1714615306952376 us; error 52 us; skew 500 ppm
I20240502 02:01:46.952803 31119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 02:01:46.960057 31119 webserver.cc:460] Webserver started at http://127.25.109.190:46755/ using document root <none> and password file <none>
I20240502 02:01:46.960930 31119 fs_manager.cc:359] Metadata directory not provided
I20240502 02:01:46.961081 31119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 02:01:46.961414 31119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 02:01:46.964375 31119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "3c13a0ed1b6d4d319f8a6bd5a55fa34c"
format_stamp: "Formatted at 2024-05-02 02:01:46 on dist-test-slave-kj4b"
I20240502 02:01:46.965257 31119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "3c13a0ed1b6d4d319f8a6bd5a55fa34c"
format_stamp: "Formatted at 2024-05-02 02:01:46 on dist-test-slave-kj4b"
I20240502 02:01:46.971061 31119 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.002s	sys 0.004s
I20240502 02:01:46.974926 31133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:01:46.976610 31119 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 02:01:46.976806 31119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal
uuid: "3c13a0ed1b6d4d319f8a6bd5a55fa34c"
format_stamp: "Formatted at 2024-05-02 02:01:46 on dist-test-slave-kj4b"
I20240502 02:01:46.977002 31119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 02:01:46.998982 31119 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 02:01:46.999954 31119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 02:01:47.000241 31119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 02:01:47.019039 31119 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.190:43455
I20240502 02:01:47.019063 31184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.190:43455 every 8 connection(s)
I20240502 02:01:47.020587 31119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 02:01:47.024516 31185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:47.026573 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 31119
I20240502 02:01:47.026779 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 02:01:47.036932 31185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c13a0ed1b6d4d319f8a6bd5a55fa34c: Bootstrap starting.
I20240502 02:01:47.040737 31185 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c13a0ed1b6d4d319f8a6bd5a55fa34c: Neither blocks nor log segments found. Creating new log.
I20240502 02:01:47.042071 31185 log.cc:824] T 00000000000000000000000000000000 P 3c13a0ed1b6d4d319f8a6bd5a55fa34c: Log is configured to *not* fsync() on all Append() calls
I20240502 02:01:47.044819 31185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c13a0ed1b6d4d319f8a6bd5a55fa34c: No bootstrap required, opened a new log
I20240502 02:01:47.051403 31185 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c13a0ed1b6d4d319f8a6bd5a55fa34c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c13a0ed1b6d4d319f8a6bd5a55fa34c" member_type: VOTER last_known_addr { host: "127.25.109.190" port: 43455 } }
I20240502 02:01:47.051838 31185 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c13a0ed1b6d4d319f8a6bd5a55fa34c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:01:47.051985 31185 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c13a0ed1b6d4d319f8a6bd5a55fa34c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c13a0ed1b6d4d319f8a6bd5a55fa34c, State: Initialized, Role: FOLLOWER
I20240502 02:01:47.052752 31185 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c13a0ed1b6d4d319f8a6bd5a55fa34c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c13a0ed1b6d4d319f8a6bd5a55fa34c" member_type: VOTER last_known_addr { host: "127.25.109.190" port: 43455 } }
I20240502 02:01:47.053051 31185 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c13a0ed1b6d4d319f8a6bd5a55fa34c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 02:01:47.053211 31185 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c13a0ed1b6d4d319f8a6bd5a55fa34c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 02:01:47.053366 31185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c13a0ed1b6d4d319f8a6bd5a55fa34c [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:01:47.055503 31185 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c13a0ed1b6d4d319f8a6bd5a55fa34c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c13a0ed1b6d4d319f8a6bd5a55fa34c" member_type: VOTER last_known_addr { host: "127.25.109.190" port: 43455 } }
I20240502 02:01:47.056073 31185 leader_election.cc:302] T 00000000000000000000000000000000 P 3c13a0ed1b6d4d319f8a6bd5a55fa34c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c13a0ed1b6d4d319f8a6bd5a55fa34c; no voters: 
I20240502 02:01:47.056658 31185 leader_election.cc:288] T 00000000000000000000000000000000 P 3c13a0ed1b6d4d319f8a6bd5a55fa34c [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 02:01:47.056814 31190 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c13a0ed1b6d4d319f8a6bd5a55fa34c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:01:47.057433 31190 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c13a0ed1b6d4d319f8a6bd5a55fa34c [term 1 LEADER]: Becoming Leader. State: Replica: 3c13a0ed1b6d4d319f8a6bd5a55fa34c, State: Running, Role: LEADER
I20240502 02:01:47.057999 31190 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c13a0ed1b6d4d319f8a6bd5a55fa34c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c13a0ed1b6d4d319f8a6bd5a55fa34c" member_type: VOTER last_known_addr { host: "127.25.109.190" port: 43455 } }
I20240502 02:01:47.058683 31185 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c13a0ed1b6d4d319f8a6bd5a55fa34c [sys.catalog]: configured and running, proceeding with master startup.
I20240502 02:01:47.060997 31191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c13a0ed1b6d4d319f8a6bd5a55fa34c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c13a0ed1b6d4d319f8a6bd5a55fa34c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c13a0ed1b6d4d319f8a6bd5a55fa34c" member_type: VOTER last_known_addr { host: "127.25.109.190" port: 43455 } } }
I20240502 02:01:47.061053 31192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c13a0ed1b6d4d319f8a6bd5a55fa34c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c13a0ed1b6d4d319f8a6bd5a55fa34c. Latest consensus state: current_term: 1 leader_uuid: "3c13a0ed1b6d4d319f8a6bd5a55fa34c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c13a0ed1b6d4d319f8a6bd5a55fa34c" member_type: VOTER last_known_addr { host: "127.25.109.190" port: 43455 } } }
I20240502 02:01:47.061563 31192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c13a0ed1b6d4d319f8a6bd5a55fa34c [sys.catalog]: This master's current role is: LEADER
I20240502 02:01:47.061582 31191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c13a0ed1b6d4d319f8a6bd5a55fa34c [sys.catalog]: This master's current role is: LEADER
I20240502 02:01:47.062624 31198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 02:01:47.068647 31198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 02:01:47.078071 31198 catalog_manager.cc:1347] Generated new cluster ID: 06a324e13dee470f9f700dab84af40ef
I20240502 02:01:47.078208 31198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 02:01:47.109146 31198 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 02:01:47.111011 31198 catalog_manager.cc:1504] Loading token signing keys...
I20240502 02:01:47.124969 31198 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 3c13a0ed1b6d4d319f8a6bd5a55fa34c: Generated new TSK 0
I20240502 02:01:47.126150 31198 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 02:01:47.137080 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.129:0
--local_ip_for_outbound_sockets=127.25.109.129
--webserver_interface=127.25.109.129
--webserver_port=0
--tserver_master_addrs=127.25.109.190:43455
--builtin_ntp_servers=127.25.109.148:42757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 02:01:47.505551 31209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:01:47.505873 31209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 02:01:47.506016 31209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:01:47.513909 31209 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 02:01:47.514063 31209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 02:01:47.514240 31209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.129
I20240502 02:01:47.523567 31209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:42757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.25.109.129
--webserver_port=0
--tserver_master_addrs=127.25.109.190:43455
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.129
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 02:01:47.524981 31209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 02:01:47.526741 31209 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 02:01:47.538695 31214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:01:47.539098 31215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:01:47.539472 31217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 02:01:47.539712 31209 server_base.cc:1034] running on GCE node
I20240502 02:01:47.540391 31209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 02:01:47.541450 31209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 02:01:47.542652 31209 hybrid_clock.cc:648] HybridClock initialized: now 1714615307542595 us; error 49 us; skew 500 ppm
I20240502 02:01:47.542979 31209 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 02:01:47.545446 31209 webserver.cc:460] Webserver started at http://127.25.109.129:44699/ using document root <none> and password file <none>
I20240502 02:01:47.546247 31209 fs_manager.cc:359] Metadata directory not provided
I20240502 02:01:47.546375 31209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 02:01:47.546713 31209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 02:01:47.549664 31209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "64688d2615e44408ab868cb531540a84"
format_stamp: "Formatted at 2024-05-02 02:01:47 on dist-test-slave-kj4b"
I20240502 02:01:47.550487 31209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "64688d2615e44408ab868cb531540a84"
format_stamp: "Formatted at 2024-05-02 02:01:47 on dist-test-slave-kj4b"
I20240502 02:01:47.556123 31209 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.001s
I20240502 02:01:47.559805 31223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:01:47.561373 31209 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 02:01:47.561558 31209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "64688d2615e44408ab868cb531540a84"
format_stamp: "Formatted at 2024-05-02 02:01:47 on dist-test-slave-kj4b"
I20240502 02:01:47.561779 31209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 02:01:47.583073 31209 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 02:01:47.584023 31209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 02:01:47.584306 31209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 02:01:47.585299 31209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 02:01:47.586824 31209 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 02:01:47.586925 31209 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:01:47.587052 31209 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 02:01:47.587121 31209 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:01:47.617789 31209 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.129:33021
I20240502 02:01:47.617848 31338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.129:33021 every 8 connection(s)
I20240502 02:01:47.619091 31209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 02:01:47.629029 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 31209
I20240502 02:01:47.629266 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 02:01:47.631544 31339 heartbeater.cc:347] Connected to a master server at 127.25.109.190:43455
I20240502 02:01:47.631879 31339 heartbeater.cc:464] Registering TS with master...
I20240502 02:01:47.632714 31339 heartbeater.cc:511] Master 127.25.109.190:43455 requested a full tablet report, sending...
I20240502 02:01:47.632870 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.130:0
--local_ip_for_outbound_sockets=127.25.109.130
--webserver_interface=127.25.109.130
--webserver_port=0
--tserver_master_addrs=127.25.109.190:43455
--builtin_ntp_servers=127.25.109.148:42757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 02:01:47.682921 31149 ts_manager.cc:194] Registered new tserver with Master: 64688d2615e44408ab868cb531540a84 (127.25.109.129:33021)
I20240502 02:01:47.684762 31149 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.129:36469
W20240502 02:01:47.978549 31344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:01:47.978863 31344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 02:01:47.979007 31344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:01:47.986817 31344 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 02:01:47.986980 31344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 02:01:47.987146 31344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.130
I20240502 02:01:47.996511 31344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:42757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.25.109.130
--webserver_port=0
--tserver_master_addrs=127.25.109.190:43455
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.130
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 02:01:47.997881 31344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 02:01:47.999711 31344 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 02:01:48.011512 31350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:01:48.011888 31349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:01:48.012281 31352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 02:01:48.012573 31344 server_base.cc:1034] running on GCE node
I20240502 02:01:48.013200 31344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 02:01:48.014185 31344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 02:01:48.015408 31344 hybrid_clock.cc:648] HybridClock initialized: now 1714615308015351 us; error 48 us; skew 500 ppm
I20240502 02:01:48.015791 31344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 02:01:48.018388 31344 webserver.cc:460] Webserver started at http://127.25.109.130:40721/ using document root <none> and password file <none>
I20240502 02:01:48.019410 31344 fs_manager.cc:359] Metadata directory not provided
I20240502 02:01:48.019565 31344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 02:01:48.020004 31344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 02:01:48.023627 31344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "0db0855dad46416094c68fc878712cbb"
format_stamp: "Formatted at 2024-05-02 02:01:48 on dist-test-slave-kj4b"
I20240502 02:01:48.024718 31344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "0db0855dad46416094c68fc878712cbb"
format_stamp: "Formatted at 2024-05-02 02:01:48 on dist-test-slave-kj4b"
I20240502 02:01:48.031934 31344 fs_manager.cc:691] Time spent creating directory manager: real 0.007s	user 0.006s	sys 0.001s
I20240502 02:01:48.036437 31358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:01:48.038081 31344 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20240502 02:01:48.038267 31344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "0db0855dad46416094c68fc878712cbb"
format_stamp: "Formatted at 2024-05-02 02:01:48 on dist-test-slave-kj4b"
I20240502 02:01:48.038533 31344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 02:01:48.071570 31344 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 02:01:48.072468 31344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 02:01:48.072721 31344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 02:01:48.073704 31344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 02:01:48.075314 31344 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 02:01:48.075404 31344 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:01:48.075500 31344 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 02:01:48.075554 31344 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:01:48.106452 31344 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.130:38025
I20240502 02:01:48.106527 31472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.130:38025 every 8 connection(s)
I20240502 02:01:48.107795 31344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 02:01:48.109392 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 31344
I20240502 02:01:48.109570 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 02:01:48.113076 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.131:0
--local_ip_for_outbound_sockets=127.25.109.131
--webserver_interface=127.25.109.131
--webserver_port=0
--tserver_master_addrs=127.25.109.190:43455
--builtin_ntp_servers=127.25.109.148:42757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 02:01:48.121732 31473 heartbeater.cc:347] Connected to a master server at 127.25.109.190:43455
I20240502 02:01:48.122114 31473 heartbeater.cc:464] Registering TS with master...
I20240502 02:01:48.123042 31473 heartbeater.cc:511] Master 127.25.109.190:43455 requested a full tablet report, sending...
I20240502 02:01:48.176681 31149 ts_manager.cc:194] Registered new tserver with Master: 0db0855dad46416094c68fc878712cbb (127.25.109.130:38025)
I20240502 02:01:48.177641 31149 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.130:59609
W20240502 02:01:48.472887 31476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:01:48.473201 31476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 02:01:48.473331 31476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:01:48.481026 31476 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 02:01:48.481166 31476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 02:01:48.481320 31476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.131
I20240502 02:01:48.490978 31476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:42757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.25.109.131
--webserver_port=0
--tserver_master_addrs=127.25.109.190:43455
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.131
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 02:01:48.492448 31476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 02:01:48.494252 31476 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 02:01:48.506192 31482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:01:48.506220 31483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 02:01:48.507660 31476 server_base.cc:1034] running on GCE node
W20240502 02:01:48.508082 31485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 02:01:48.508744 31476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 02:01:48.509690 31476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 02:01:48.510900 31476 hybrid_clock.cc:648] HybridClock initialized: now 1714615308510837 us; error 60 us; skew 500 ppm
I20240502 02:01:48.511229 31476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 02:01:48.513546 31476 webserver.cc:460] Webserver started at http://127.25.109.131:34345/ using document root <none> and password file <none>
I20240502 02:01:48.514321 31476 fs_manager.cc:359] Metadata directory not provided
I20240502 02:01:48.514447 31476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 02:01:48.514750 31476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 02:01:48.517503 31476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "e325c0c7447f4b329994aafdee580666"
format_stamp: "Formatted at 2024-05-02 02:01:48 on dist-test-slave-kj4b"
I20240502 02:01:48.518271 31476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "e325c0c7447f4b329994aafdee580666"
format_stamp: "Formatted at 2024-05-02 02:01:48 on dist-test-slave-kj4b"
I20240502 02:01:48.523589 31476 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20240502 02:01:48.527047 31491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:01:48.528486 31476 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.003s
I20240502 02:01:48.528656 31476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "e325c0c7447f4b329994aafdee580666"
format_stamp: "Formatted at 2024-05-02 02:01:48 on dist-test-slave-kj4b"
I20240502 02:01:48.528872 31476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 02:01:48.557476 31476 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 02:01:48.558409 31476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 02:01:48.558686 31476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 02:01:48.559754 31476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 02:01:48.561412 31476 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 02:01:48.561517 31476 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:01:48.561630 31476 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 02:01:48.561702 31476 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:01:48.591877 31476 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.131:43937
I20240502 02:01:48.591965 31605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.131:43937 every 8 connection(s)
I20240502 02:01:48.593171 31476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 02:01:48.597393 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 31476
I20240502 02:01:48.597617 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 02:01:48.601358 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.132:0
--local_ip_for_outbound_sockets=127.25.109.132
--webserver_interface=127.25.109.132
--webserver_port=0
--tserver_master_addrs=127.25.109.190:43455
--builtin_ntp_servers=127.25.109.148:42757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 02:01:48.606971 31606 heartbeater.cc:347] Connected to a master server at 127.25.109.190:43455
I20240502 02:01:48.607287 31606 heartbeater.cc:464] Registering TS with master...
I20240502 02:01:48.608256 31606 heartbeater.cc:511] Master 127.25.109.190:43455 requested a full tablet report, sending...
I20240502 02:01:48.651489 31149 ts_manager.cc:194] Registered new tserver with Master: e325c0c7447f4b329994aafdee580666 (127.25.109.131:43937)
I20240502 02:01:48.652482 31149 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.131:53025
I20240502 02:01:48.688493 31339 heartbeater.cc:503] Master 127.25.109.190:43455 was elected leader, sending a full tablet report...
W20240502 02:01:48.947253 31611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:01:48.947562 31611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 02:01:48.947664 31611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:01:48.955395 31611 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 02:01:48.955546 31611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 02:01:48.955706 31611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.132
I20240502 02:01:48.965082 31611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:42757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.25.109.132
--webserver_port=0
--tserver_master_addrs=127.25.109.190:43455
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.132
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 02:01:48.966475 31611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 02:01:48.968353 31611 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 02:01:48.980000 31619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:01:48.980305 31617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:01:48.980758 31616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 02:01:48.981072 31611 server_base.cc:1034] running on GCE node
I20240502 02:01:48.981673 31611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 02:01:48.982646 31611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 02:01:48.983850 31611 hybrid_clock.cc:648] HybridClock initialized: now 1714615308983798 us; error 48 us; skew 500 ppm
I20240502 02:01:48.984218 31611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 02:01:48.986445 31611 webserver.cc:460] Webserver started at http://127.25.109.132:35619/ using document root <none> and password file <none>
I20240502 02:01:48.987211 31611 fs_manager.cc:359] Metadata directory not provided
I20240502 02:01:48.987331 31611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 02:01:48.987640 31611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 02:01:48.990424 31611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "29755ed6e9444b2daf5807bd23451d29"
format_stamp: "Formatted at 2024-05-02 02:01:48 on dist-test-slave-kj4b"
I20240502 02:01:48.991204 31611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "29755ed6e9444b2daf5807bd23451d29"
format_stamp: "Formatted at 2024-05-02 02:01:48 on dist-test-slave-kj4b"
I20240502 02:01:48.996616 31611 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
I20240502 02:01:49.000222 31625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:01:49.001605 31611 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20240502 02:01:49.001776 31611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "29755ed6e9444b2daf5807bd23451d29"
format_stamp: "Formatted at 2024-05-02 02:01:48 on dist-test-slave-kj4b"
I20240502 02:01:49.001996 31611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 02:01:49.027573 31611 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 02:01:49.028739 31611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 02:01:49.029080 31611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 02:01:49.030095 31611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 02:01:49.031752 31611 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 02:01:49.031841 31611 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:01:49.031975 31611 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 02:01:49.032042 31611 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:01:49.062492 31611 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.132:38023
I20240502 02:01:49.062537 31741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.132:38023 every 8 connection(s)
I20240502 02:01:49.063845 31611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 02:01:49.070819 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 31611
I20240502 02:01:49.070983 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 02:01:49.074476 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.133:0
--local_ip_for_outbound_sockets=127.25.109.133
--webserver_interface=127.25.109.133
--webserver_port=0
--tserver_master_addrs=127.25.109.190:43455
--builtin_ntp_servers=127.25.109.148:42757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 02:01:49.077358 31742 heartbeater.cc:347] Connected to a master server at 127.25.109.190:43455
I20240502 02:01:49.077690 31742 heartbeater.cc:464] Registering TS with master...
I20240502 02:01:49.078537 31742 heartbeater.cc:511] Master 127.25.109.190:43455 requested a full tablet report, sending...
I20240502 02:01:49.127776 31149 ts_manager.cc:194] Registered new tserver with Master: 29755ed6e9444b2daf5807bd23451d29 (127.25.109.132:38023)
I20240502 02:01:49.129137 31149 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.132:48995
I20240502 02:01:49.180393 31473 heartbeater.cc:503] Master 127.25.109.190:43455 was elected leader, sending a full tablet report...
W20240502 02:01:49.429746 31746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:01:49.430070 31746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 02:01:49.430199 31746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:01:49.437883 31746 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 02:01:49.438035 31746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 02:01:49.438194 31746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.133
I20240502 02:01:49.447458 31746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:42757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.25.109.133
--webserver_port=0
--tserver_master_addrs=127.25.109.190:43455
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.133
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 02:01:49.448818 31746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 02:01:49.450506 31746 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 02:01:49.461997 31751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:01:49.462409 31752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:01:49.462699 31754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 02:01:49.463071 31746 server_base.cc:1034] running on GCE node
I20240502 02:01:49.463708 31746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 02:01:49.464744 31746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 02:01:49.465953 31746 hybrid_clock.cc:648] HybridClock initialized: now 1714615309465904 us; error 40 us; skew 500 ppm
I20240502 02:01:49.466344 31746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 02:01:49.468815 31746 webserver.cc:460] Webserver started at http://127.25.109.133:37013/ using document root <none> and password file <none>
I20240502 02:01:49.469583 31746 fs_manager.cc:359] Metadata directory not provided
I20240502 02:01:49.469707 31746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 02:01:49.470028 31746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 02:01:49.472796 31746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "7da2c0838fce40c5b5000c4d71c72708"
format_stamp: "Formatted at 2024-05-02 02:01:49 on dist-test-slave-kj4b"
I20240502 02:01:49.473601 31746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "7da2c0838fce40c5b5000c4d71c72708"
format_stamp: "Formatted at 2024-05-02 02:01:49 on dist-test-slave-kj4b"
I20240502 02:01:49.479142 31746 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20240502 02:01:49.482995 31760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:01:49.484395 31746 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.002s
I20240502 02:01:49.484571 31746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "7da2c0838fce40c5b5000c4d71c72708"
format_stamp: "Formatted at 2024-05-02 02:01:49 on dist-test-slave-kj4b"
I20240502 02:01:49.484789 31746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 02:01:49.514961 31746 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 02:01:49.515895 31746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 02:01:49.516212 31746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 02:01:49.517213 31746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 02:01:49.518739 31746 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 02:01:49.518834 31746 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:01:49.518950 31746 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 02:01:49.519021 31746 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:01:49.549365 31746 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.133:36143
I20240502 02:01:49.549429 31874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.133:36143 every 8 connection(s)
I20240502 02:01:49.550720 31746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 02:01:49.557971 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 31746
I20240502 02:01:49.558156 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 02:01:49.563985 31875 heartbeater.cc:347] Connected to a master server at 127.25.109.190:43455
I20240502 02:01:49.564286 31875 heartbeater.cc:464] Registering TS with master...
I20240502 02:01:49.565128 31875 heartbeater.cc:511] Master 127.25.109.190:43455 requested a full tablet report, sending...
I20240502 02:01:49.610946 31149 ts_manager.cc:194] Registered new tserver with Master: 7da2c0838fce40c5b5000c4d71c72708 (127.25.109.133:36143)
I20240502 02:01:49.612100 31149 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.133:59451
I20240502 02:01:49.613543 26038 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 02:01:49.630543 26038 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 02:01:49.630712 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 31611
I20240502 02:01:49.638787 31737 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 02:01:49.655308 31606 heartbeater.cc:503] Master 127.25.109.190:43455 was elected leader, sending a full tablet report...
I20240502 02:01:49.721151 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 31611
I20240502 02:01:49.737502 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 31746
I20240502 02:01:49.746285 31870 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 02:01:49.827908 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 31746
I20240502 02:01:53.596016 31149 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52116:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240502 02:01:53.603040 31149 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 02:01:53.637849 31406 tablet_service.cc:1467] Processing CreateTablet for tablet 28de9be2be73409387480db617b0031c (DEFAULT_TABLE table=rebalance_test_table_0 [id=c8cf2d1749644bbb8b2330bcbf7cb196]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 02:01:53.639178 31272 tablet_service.cc:1467] Processing CreateTablet for tablet bec2e393d5e841c7857d6fd092aac53e (DEFAULT_TABLE table=rebalance_test_table_0 [id=c8cf2d1749644bbb8b2330bcbf7cb196]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 02:01:53.639711 31270 tablet_service.cc:1467] Processing CreateTablet for tablet 3475ca93c949435699b590bd97e3735c (DEFAULT_TABLE table=rebalance_test_table_0 [id=c8cf2d1749644bbb8b2330bcbf7cb196]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 02:01:53.640204 31271 tablet_service.cc:1467] Processing CreateTablet for tablet 28de9be2be73409387480db617b0031c (DEFAULT_TABLE table=rebalance_test_table_0 [id=c8cf2d1749644bbb8b2330bcbf7cb196]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 02:01:53.640836 31270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3475ca93c949435699b590bd97e3735c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:53.641345 31271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28de9be2be73409387480db617b0031c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:53.641371 31406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28de9be2be73409387480db617b0031c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:53.639039 31405 tablet_service.cc:1467] Processing CreateTablet for tablet 3475ca93c949435699b590bd97e3735c (DEFAULT_TABLE table=rebalance_test_table_0 [id=c8cf2d1749644bbb8b2330bcbf7cb196]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 02:01:53.642776 31272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bec2e393d5e841c7857d6fd092aac53e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:53.637809 31407 tablet_service.cc:1467] Processing CreateTablet for tablet bec2e393d5e841c7857d6fd092aac53e (DEFAULT_TABLE table=rebalance_test_table_0 [id=c8cf2d1749644bbb8b2330bcbf7cb196]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 02:01:53.643532 31405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3475ca93c949435699b590bd97e3735c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:53.645424 31407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bec2e393d5e841c7857d6fd092aac53e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:53.645120 31539 tablet_service.cc:1467] Processing CreateTablet for tablet bec2e393d5e841c7857d6fd092aac53e (DEFAULT_TABLE table=rebalance_test_table_0 [id=c8cf2d1749644bbb8b2330bcbf7cb196]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 02:01:53.645229 31537 tablet_service.cc:1467] Processing CreateTablet for tablet 3475ca93c949435699b590bd97e3735c (DEFAULT_TABLE table=rebalance_test_table_0 [id=c8cf2d1749644bbb8b2330bcbf7cb196]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 02:01:53.645442 31538 tablet_service.cc:1467] Processing CreateTablet for tablet 28de9be2be73409387480db617b0031c (DEFAULT_TABLE table=rebalance_test_table_0 [id=c8cf2d1749644bbb8b2330bcbf7cb196]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 02:01:53.646693 31539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bec2e393d5e841c7857d6fd092aac53e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:53.647154 31537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3475ca93c949435699b590bd97e3735c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:53.647740 31538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28de9be2be73409387480db617b0031c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:53.671761 31901 tablet_bootstrap.cc:492] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb: Bootstrap starting.
I20240502 02:01:53.675956 31899 tablet_bootstrap.cc:492] T 28de9be2be73409387480db617b0031c P 64688d2615e44408ab868cb531540a84: Bootstrap starting.
I20240502 02:01:53.677606 31901 tablet_bootstrap.cc:654] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb: Neither blocks nor log segments found. Creating new log.
I20240502 02:01:53.677697 31900 tablet_bootstrap.cc:492] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666: Bootstrap starting.
I20240502 02:01:53.679270 31899 tablet_bootstrap.cc:654] T 28de9be2be73409387480db617b0031c P 64688d2615e44408ab868cb531540a84: Neither blocks nor log segments found. Creating new log.
I20240502 02:01:53.679234 31901 log.cc:824] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb: Log is configured to *not* fsync() on all Append() calls
I20240502 02:01:53.680749 31899 log.cc:824] T 28de9be2be73409387480db617b0031c P 64688d2615e44408ab868cb531540a84: Log is configured to *not* fsync() on all Append() calls
I20240502 02:01:53.681447 31900 tablet_bootstrap.cc:654] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666: Neither blocks nor log segments found. Creating new log.
I20240502 02:01:53.682740 31901 tablet_bootstrap.cc:492] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb: No bootstrap required, opened a new log
I20240502 02:01:53.682825 31900 log.cc:824] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666: Log is configured to *not* fsync() on all Append() calls
I20240502 02:01:53.683132 31901 ts_tablet_manager.cc:1367] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb: Time spent bootstrapping tablet: real 0.012s	user 0.006s	sys 0.003s
I20240502 02:01:53.683553 31899 tablet_bootstrap.cc:492] T 28de9be2be73409387480db617b0031c P 64688d2615e44408ab868cb531540a84: No bootstrap required, opened a new log
I20240502 02:01:53.683845 31899 ts_tablet_manager.cc:1367] T 28de9be2be73409387480db617b0031c P 64688d2615e44408ab868cb531540a84: Time spent bootstrapping tablet: real 0.008s	user 0.007s	sys 0.000s
I20240502 02:01:53.685782 31900 tablet_bootstrap.cc:492] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666: No bootstrap required, opened a new log
I20240502 02:01:53.686056 31900 ts_tablet_manager.cc:1367] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666: Time spent bootstrapping tablet: real 0.009s	user 0.006s	sys 0.000s
I20240502 02:01:53.690089 31899 raft_consensus.cc:348] T 28de9be2be73409387480db617b0031c P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:53.690555 31899 raft_consensus.cc:374] T 28de9be2be73409387480db617b0031c P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:01:53.690676 31899 raft_consensus.cc:729] T 28de9be2be73409387480db617b0031c P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64688d2615e44408ab868cb531540a84, State: Initialized, Role: FOLLOWER
I20240502 02:01:53.691507 31899 consensus_queue.cc:260] T 28de9be2be73409387480db617b0031c P 64688d2615e44408ab868cb531540a84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:53.691821 31901 raft_consensus.cc:348] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:53.692370 31901 raft_consensus.cc:374] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:01:53.692516 31901 raft_consensus.cc:729] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0db0855dad46416094c68fc878712cbb, State: Initialized, Role: FOLLOWER
I20240502 02:01:53.693199 31901 consensus_queue.cc:260] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:53.694356 31899 ts_tablet_manager.cc:1398] T 28de9be2be73409387480db617b0031c P 64688d2615e44408ab868cb531540a84: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.003s
I20240502 02:01:53.694933 31899 tablet_bootstrap.cc:492] T bec2e393d5e841c7857d6fd092aac53e P 64688d2615e44408ab868cb531540a84: Bootstrap starting.
I20240502 02:01:53.695307 31900 raft_consensus.cc:348] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:53.695818 31900 raft_consensus.cc:374] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:01:53.695978 31900 raft_consensus.cc:729] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e325c0c7447f4b329994aafdee580666, State: Initialized, Role: FOLLOWER
I20240502 02:01:53.696873 31900 consensus_queue.cc:260] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:53.697079 31899 tablet_bootstrap.cc:654] T bec2e393d5e841c7857d6fd092aac53e P 64688d2615e44408ab868cb531540a84: Neither blocks nor log segments found. Creating new log.
I20240502 02:01:53.701732 31901 ts_tablet_manager.cc:1398] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb: Time spent starting tablet: real 0.018s	user 0.012s	sys 0.004s
I20240502 02:01:53.702349 31901 tablet_bootstrap.cc:492] T bec2e393d5e841c7857d6fd092aac53e P 0db0855dad46416094c68fc878712cbb: Bootstrap starting.
I20240502 02:01:53.704417 31901 tablet_bootstrap.cc:654] T bec2e393d5e841c7857d6fd092aac53e P 0db0855dad46416094c68fc878712cbb: Neither blocks nor log segments found. Creating new log.
I20240502 02:01:53.706224 31907 raft_consensus.cc:482] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 02:01:53.706602 31907 raft_consensus.cc:504] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:53.708963 31901 tablet_bootstrap.cc:492] T bec2e393d5e841c7857d6fd092aac53e P 0db0855dad46416094c68fc878712cbb: No bootstrap required, opened a new log
I20240502 02:01:53.709162 31900 ts_tablet_manager.cc:1398] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666: Time spent starting tablet: real 0.023s	user 0.010s	sys 0.008s
I20240502 02:01:53.709185 31901 ts_tablet_manager.cc:1367] T bec2e393d5e841c7857d6fd092aac53e P 0db0855dad46416094c68fc878712cbb: Time spent bootstrapping tablet: real 0.007s	user 0.004s	sys 0.000s
I20240502 02:01:53.709681 31900 tablet_bootstrap.cc:492] T bec2e393d5e841c7857d6fd092aac53e P e325c0c7447f4b329994aafdee580666: Bootstrap starting.
I20240502 02:01:53.710336 31901 raft_consensus.cc:348] T bec2e393d5e841c7857d6fd092aac53e P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:53.710655 31901 raft_consensus.cc:374] T bec2e393d5e841c7857d6fd092aac53e P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:01:53.710815 31901 raft_consensus.cc:729] T bec2e393d5e841c7857d6fd092aac53e P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0db0855dad46416094c68fc878712cbb, State: Initialized, Role: FOLLOWER
I20240502 02:01:53.711115 31901 consensus_queue.cc:260] T bec2e393d5e841c7857d6fd092aac53e P 0db0855dad46416094c68fc878712cbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:53.711565 31900 tablet_bootstrap.cc:654] T bec2e393d5e841c7857d6fd092aac53e P e325c0c7447f4b329994aafdee580666: Neither blocks nor log segments found. Creating new log.
I20240502 02:01:53.711951 31901 ts_tablet_manager.cc:1398] T bec2e393d5e841c7857d6fd092aac53e P 0db0855dad46416094c68fc878712cbb: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 02:01:53.712704 31901 tablet_bootstrap.cc:492] T 3475ca93c949435699b590bd97e3735c P 0db0855dad46416094c68fc878712cbb: Bootstrap starting.
I20240502 02:01:53.714578 31901 tablet_bootstrap.cc:654] T 3475ca93c949435699b590bd97e3735c P 0db0855dad46416094c68fc878712cbb: Neither blocks nor log segments found. Creating new log.
I20240502 02:01:53.716524 31899 tablet_bootstrap.cc:492] T bec2e393d5e841c7857d6fd092aac53e P 64688d2615e44408ab868cb531540a84: No bootstrap required, opened a new log
I20240502 02:01:53.717759 31899 ts_tablet_manager.cc:1367] T bec2e393d5e841c7857d6fd092aac53e P 64688d2615e44408ab868cb531540a84: Time spent bootstrapping tablet: real 0.023s	user 0.009s	sys 0.004s
I20240502 02:01:53.720221 31907 leader_election.cc:288] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64688d2615e44408ab868cb531540a84 (127.25.109.129:33021), 0db0855dad46416094c68fc878712cbb (127.25.109.130:38025)
I20240502 02:01:53.720850 31899 raft_consensus.cc:348] T bec2e393d5e841c7857d6fd092aac53e P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:53.721699 31899 raft_consensus.cc:374] T bec2e393d5e841c7857d6fd092aac53e P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:01:53.721982 31899 raft_consensus.cc:729] T bec2e393d5e841c7857d6fd092aac53e P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64688d2615e44408ab868cb531540a84, State: Initialized, Role: FOLLOWER
I20240502 02:01:53.722268 31899 consensus_queue.cc:260] T bec2e393d5e841c7857d6fd092aac53e P 64688d2615e44408ab868cb531540a84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:53.723641 31899 ts_tablet_manager.cc:1398] T bec2e393d5e841c7857d6fd092aac53e P 64688d2615e44408ab868cb531540a84: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240502 02:01:53.724927 31899 tablet_bootstrap.cc:492] T 3475ca93c949435699b590bd97e3735c P 64688d2615e44408ab868cb531540a84: Bootstrap starting.
I20240502 02:01:53.725597 31428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3475ca93c949435699b590bd97e3735c" candidate_uuid: "e325c0c7447f4b329994aafdee580666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0db0855dad46416094c68fc878712cbb" is_pre_election: true
I20240502 02:01:53.727356 31899 tablet_bootstrap.cc:654] T 3475ca93c949435699b590bd97e3735c P 64688d2615e44408ab868cb531540a84: Neither blocks nor log segments found. Creating new log.
W20240502 02:01:53.727224 31495 leader_election.cc:341] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0db0855dad46416094c68fc878712cbb (127.25.109.130:38025): Illegal state: must be running to vote when last-logged opid is not known
I20240502 02:01:53.725597 31901 tablet_bootstrap.cc:492] T 3475ca93c949435699b590bd97e3735c P 0db0855dad46416094c68fc878712cbb: No bootstrap required, opened a new log
I20240502 02:01:53.727720 31901 ts_tablet_manager.cc:1367] T 3475ca93c949435699b590bd97e3735c P 0db0855dad46416094c68fc878712cbb: Time spent bootstrapping tablet: real 0.015s	user 0.002s	sys 0.003s
I20240502 02:01:53.728626 31901 raft_consensus.cc:348] T 3475ca93c949435699b590bd97e3735c P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:53.728907 31901 raft_consensus.cc:374] T 3475ca93c949435699b590bd97e3735c P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:01:53.728986 31901 raft_consensus.cc:729] T 3475ca93c949435699b590bd97e3735c P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0db0855dad46416094c68fc878712cbb, State: Initialized, Role: FOLLOWER
I20240502 02:01:53.729264 31901 consensus_queue.cc:260] T 3475ca93c949435699b590bd97e3735c P 0db0855dad46416094c68fc878712cbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:53.730109 31901 ts_tablet_manager.cc:1398] T 3475ca93c949435699b590bd97e3735c P 0db0855dad46416094c68fc878712cbb: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 02:01:53.731913 31900 tablet_bootstrap.cc:492] T bec2e393d5e841c7857d6fd092aac53e P e325c0c7447f4b329994aafdee580666: No bootstrap required, opened a new log
I20240502 02:01:53.732149 31900 ts_tablet_manager.cc:1367] T bec2e393d5e841c7857d6fd092aac53e P e325c0c7447f4b329994aafdee580666: Time spent bootstrapping tablet: real 0.023s	user 0.009s	sys 0.013s
I20240502 02:01:53.733381 31900 raft_consensus.cc:348] T bec2e393d5e841c7857d6fd092aac53e P e325c0c7447f4b329994aafdee580666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:53.733743 31900 raft_consensus.cc:374] T bec2e393d5e841c7857d6fd092aac53e P e325c0c7447f4b329994aafdee580666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:01:53.733937 31900 raft_consensus.cc:729] T bec2e393d5e841c7857d6fd092aac53e P e325c0c7447f4b329994aafdee580666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e325c0c7447f4b329994aafdee580666, State: Initialized, Role: FOLLOWER
I20240502 02:01:53.734256 31900 consensus_queue.cc:260] T bec2e393d5e841c7857d6fd092aac53e P e325c0c7447f4b329994aafdee580666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:53.734982 31900 ts_tablet_manager.cc:1398] T bec2e393d5e841c7857d6fd092aac53e P e325c0c7447f4b329994aafdee580666: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 02:01:53.735348 31900 tablet_bootstrap.cc:492] T 28de9be2be73409387480db617b0031c P e325c0c7447f4b329994aafdee580666: Bootstrap starting.
I20240502 02:01:53.736589 31899 tablet_bootstrap.cc:492] T 3475ca93c949435699b590bd97e3735c P 64688d2615e44408ab868cb531540a84: No bootstrap required, opened a new log
I20240502 02:01:53.736871 31899 ts_tablet_manager.cc:1367] T 3475ca93c949435699b590bd97e3735c P 64688d2615e44408ab868cb531540a84: Time spent bootstrapping tablet: real 0.012s	user 0.005s	sys 0.005s
I20240502 02:01:53.737120 31900 tablet_bootstrap.cc:654] T 28de9be2be73409387480db617b0031c P e325c0c7447f4b329994aafdee580666: Neither blocks nor log segments found. Creating new log.
I20240502 02:01:53.737795 31899 raft_consensus.cc:348] T 3475ca93c949435699b590bd97e3735c P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:53.738147 31899 raft_consensus.cc:374] T 3475ca93c949435699b590bd97e3735c P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:01:53.738298 31899 raft_consensus.cc:729] T 3475ca93c949435699b590bd97e3735c P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64688d2615e44408ab868cb531540a84, State: Initialized, Role: FOLLOWER
I20240502 02:01:53.738605 31899 consensus_queue.cc:260] T 3475ca93c949435699b590bd97e3735c P 64688d2615e44408ab868cb531540a84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:53.738927 31293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3475ca93c949435699b590bd97e3735c" candidate_uuid: "e325c0c7447f4b329994aafdee580666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64688d2615e44408ab868cb531540a84" is_pre_election: true
I20240502 02:01:53.739212 31906 raft_consensus.cc:482] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 02:01:53.739609 31293 raft_consensus.cc:2454] T 3475ca93c949435699b590bd97e3735c P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e325c0c7447f4b329994aafdee580666 in term 0.
I20240502 02:01:53.739464 31906 raft_consensus.cc:504] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:53.740311 31494 leader_election.cc:302] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 64688d2615e44408ab868cb531540a84, e325c0c7447f4b329994aafdee580666; no voters: 0db0855dad46416094c68fc878712cbb
I20240502 02:01:53.740442 31899 ts_tablet_manager.cc:1398] T 3475ca93c949435699b590bd97e3735c P 64688d2615e44408ab868cb531540a84: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 02:01:53.740784 31907 raft_consensus.cc:2781] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 02:01:53.740937 31907 raft_consensus.cc:482] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 02:01:53.741050 31907 raft_consensus.cc:3037] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:01:53.741514 31900 tablet_bootstrap.cc:492] T 28de9be2be73409387480db617b0031c P e325c0c7447f4b329994aafdee580666: No bootstrap required, opened a new log
I20240502 02:01:53.741693 31900 ts_tablet_manager.cc:1367] T 28de9be2be73409387480db617b0031c P e325c0c7447f4b329994aafdee580666: Time spent bootstrapping tablet: real 0.006s	user 0.003s	sys 0.000s
I20240502 02:01:53.741686 31906 leader_election.cc:288] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64688d2615e44408ab868cb531540a84 (127.25.109.129:33021), e325c0c7447f4b329994aafdee580666 (127.25.109.131:43937)
I20240502 02:01:53.742480 31900 raft_consensus.cc:348] T 28de9be2be73409387480db617b0031c P e325c0c7447f4b329994aafdee580666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:53.742717 31900 raft_consensus.cc:374] T 28de9be2be73409387480db617b0031c P e325c0c7447f4b329994aafdee580666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:01:53.742803 31900 raft_consensus.cc:729] T 28de9be2be73409387480db617b0031c P e325c0c7447f4b329994aafdee580666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e325c0c7447f4b329994aafdee580666, State: Initialized, Role: FOLLOWER
I20240502 02:01:53.743062 31900 consensus_queue.cc:260] T 28de9be2be73409387480db617b0031c P e325c0c7447f4b329994aafdee580666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:53.744130 31900 ts_tablet_manager.cc:1398] T 28de9be2be73409387480db617b0031c P e325c0c7447f4b329994aafdee580666: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.003s
I20240502 02:01:53.744618 31907 raft_consensus.cc:504] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:53.745290 31907 leader_election.cc:288] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 [CANDIDATE]: Term 1 election: Requested vote from peers 64688d2615e44408ab868cb531540a84 (127.25.109.129:33021), 0db0855dad46416094c68fc878712cbb (127.25.109.130:38025)
I20240502 02:01:53.745877 31293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3475ca93c949435699b590bd97e3735c" candidate_uuid: "e325c0c7447f4b329994aafdee580666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64688d2615e44408ab868cb531540a84"
I20240502 02:01:53.746105 31293 raft_consensus.cc:3037] T 3475ca93c949435699b590bd97e3735c P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:01:53.746450 31428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3475ca93c949435699b590bd97e3735c" candidate_uuid: "e325c0c7447f4b329994aafdee580666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0db0855dad46416094c68fc878712cbb"
I20240502 02:01:53.746692 31428 raft_consensus.cc:3037] T 3475ca93c949435699b590bd97e3735c P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:01:53.749157 31293 raft_consensus.cc:2454] T 3475ca93c949435699b590bd97e3735c P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e325c0c7447f4b329994aafdee580666 in term 1.
I20240502 02:01:53.749703 31494 leader_election.cc:302] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64688d2615e44408ab868cb531540a84, e325c0c7447f4b329994aafdee580666; no voters: 
I20240502 02:01:53.749816 31428 raft_consensus.cc:2454] T 3475ca93c949435699b590bd97e3735c P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e325c0c7447f4b329994aafdee580666 in term 1.
I20240502 02:01:53.750077 31907 raft_consensus.cc:2781] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:01:53.750250 31907 raft_consensus.cc:686] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 [term 1 LEADER]: Becoming Leader. State: Replica: e325c0c7447f4b329994aafdee580666, State: Running, Role: LEADER
I20240502 02:01:53.750761 31907 consensus_queue.cc:237] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:53.753295 31293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28de9be2be73409387480db617b0031c" candidate_uuid: "0db0855dad46416094c68fc878712cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64688d2615e44408ab868cb531540a84" is_pre_election: true
I20240502 02:01:53.753542 31293 raft_consensus.cc:2454] T 28de9be2be73409387480db617b0031c P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0db0855dad46416094c68fc878712cbb in term 0.
I20240502 02:01:53.754120 31361 leader_election.cc:302] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0db0855dad46416094c68fc878712cbb, 64688d2615e44408ab868cb531540a84; no voters: 
I20240502 02:01:53.754609 31906 raft_consensus.cc:2781] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 02:01:53.754743 31906 raft_consensus.cc:482] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 02:01:53.754827 31906 raft_consensus.cc:3037] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:01:53.755066 31560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28de9be2be73409387480db617b0031c" candidate_uuid: "0db0855dad46416094c68fc878712cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e325c0c7447f4b329994aafdee580666" is_pre_election: true
I20240502 02:01:53.755491 31560 raft_consensus.cc:2454] T 28de9be2be73409387480db617b0031c P e325c0c7447f4b329994aafdee580666 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0db0855dad46416094c68fc878712cbb in term 0.
I20240502 02:01:53.756271 31920 raft_consensus.cc:482] T bec2e393d5e841c7857d6fd092aac53e P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 02:01:53.756542 31920 raft_consensus.cc:504] T bec2e393d5e841c7857d6fd092aac53e P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:53.757428 31920 leader_election.cc:288] T bec2e393d5e841c7857d6fd092aac53e P 0db0855dad46416094c68fc878712cbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64688d2615e44408ab868cb531540a84 (127.25.109.129:33021), e325c0c7447f4b329994aafdee580666 (127.25.109.131:43937)
I20240502 02:01:53.757817 31906 raft_consensus.cc:504] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:53.757232 31150 catalog_manager.cc:5484] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 reported cstate change: term changed from 0 to 1, leader changed from <none> to e325c0c7447f4b329994aafdee580666 (127.25.109.131). New cstate: current_term: 1 leader_uuid: "e325c0c7447f4b329994aafdee580666" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } health_report { overall_health: HEALTHY } } }
I20240502 02:01:53.758056 31560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec2e393d5e841c7857d6fd092aac53e" candidate_uuid: "0db0855dad46416094c68fc878712cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e325c0c7447f4b329994aafdee580666" is_pre_election: true
I20240502 02:01:53.758260 31560 raft_consensus.cc:2454] T bec2e393d5e841c7857d6fd092aac53e P e325c0c7447f4b329994aafdee580666 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0db0855dad46416094c68fc878712cbb in term 0.
I20240502 02:01:53.758299 31293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec2e393d5e841c7857d6fd092aac53e" candidate_uuid: "0db0855dad46416094c68fc878712cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64688d2615e44408ab868cb531540a84" is_pre_election: true
I20240502 02:01:53.758554 31293 raft_consensus.cc:2454] T bec2e393d5e841c7857d6fd092aac53e P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0db0855dad46416094c68fc878712cbb in term 0.
I20240502 02:01:53.758522 31906 leader_election.cc:288] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb [CANDIDATE]: Term 1 election: Requested vote from peers 64688d2615e44408ab868cb531540a84 (127.25.109.129:33021), e325c0c7447f4b329994aafdee580666 (127.25.109.131:43937)
I20240502 02:01:53.759042 31293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28de9be2be73409387480db617b0031c" candidate_uuid: "0db0855dad46416094c68fc878712cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64688d2615e44408ab868cb531540a84"
I20240502 02:01:53.759150 31361 leader_election.cc:302] T bec2e393d5e841c7857d6fd092aac53e P 0db0855dad46416094c68fc878712cbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0db0855dad46416094c68fc878712cbb, 64688d2615e44408ab868cb531540a84; no voters: 
I20240502 02:01:53.759161 31560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28de9be2be73409387480db617b0031c" candidate_uuid: "0db0855dad46416094c68fc878712cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e325c0c7447f4b329994aafdee580666"
I20240502 02:01:53.759243 31293 raft_consensus.cc:3037] T 28de9be2be73409387480db617b0031c P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:01:53.759344 31560 raft_consensus.cc:3037] T 28de9be2be73409387480db617b0031c P e325c0c7447f4b329994aafdee580666 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:01:53.759645 31906 raft_consensus.cc:2781] T bec2e393d5e841c7857d6fd092aac53e P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 02:01:53.759817 31906 raft_consensus.cc:482] T bec2e393d5e841c7857d6fd092aac53e P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 02:01:53.759976 31906 raft_consensus.cc:3037] T bec2e393d5e841c7857d6fd092aac53e P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:01:53.761610 31560 raft_consensus.cc:2454] T 28de9be2be73409387480db617b0031c P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0db0855dad46416094c68fc878712cbb in term 1.
I20240502 02:01:53.762113 31361 leader_election.cc:302] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0db0855dad46416094c68fc878712cbb, e325c0c7447f4b329994aafdee580666; no voters: 
I20240502 02:01:53.762112 31293 raft_consensus.cc:2454] T 28de9be2be73409387480db617b0031c P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0db0855dad46416094c68fc878712cbb in term 1.
I20240502 02:01:53.762454 31920 raft_consensus.cc:2781] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:01:53.762786 31906 raft_consensus.cc:504] T bec2e393d5e841c7857d6fd092aac53e P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:53.763037 31920 raft_consensus.cc:686] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb [term 1 LEADER]: Becoming Leader. State: Replica: 0db0855dad46416094c68fc878712cbb, State: Running, Role: LEADER
I20240502 02:01:53.763809 31920 consensus_queue.cc:237] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:53.764480 31293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec2e393d5e841c7857d6fd092aac53e" candidate_uuid: "0db0855dad46416094c68fc878712cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64688d2615e44408ab868cb531540a84"
I20240502 02:01:53.764714 31293 raft_consensus.cc:3037] T bec2e393d5e841c7857d6fd092aac53e P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:01:53.765722 31560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec2e393d5e841c7857d6fd092aac53e" candidate_uuid: "0db0855dad46416094c68fc878712cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e325c0c7447f4b329994aafdee580666"
I20240502 02:01:53.765975 31560 raft_consensus.cc:3037] T bec2e393d5e841c7857d6fd092aac53e P e325c0c7447f4b329994aafdee580666 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:01:53.768049 31906 leader_election.cc:288] T bec2e393d5e841c7857d6fd092aac53e P 0db0855dad46416094c68fc878712cbb [CANDIDATE]: Term 1 election: Requested vote from peers 64688d2615e44408ab868cb531540a84 (127.25.109.129:33021), e325c0c7447f4b329994aafdee580666 (127.25.109.131:43937)
I20240502 02:01:53.768357 31293 raft_consensus.cc:2454] T bec2e393d5e841c7857d6fd092aac53e P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0db0855dad46416094c68fc878712cbb in term 1.
I20240502 02:01:53.768844 31560 raft_consensus.cc:2454] T bec2e393d5e841c7857d6fd092aac53e P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0db0855dad46416094c68fc878712cbb in term 1.
I20240502 02:01:53.768889 31361 leader_election.cc:302] T bec2e393d5e841c7857d6fd092aac53e P 0db0855dad46416094c68fc878712cbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0db0855dad46416094c68fc878712cbb, 64688d2615e44408ab868cb531540a84; no voters: 
I20240502 02:01:53.769205 31906 raft_consensus.cc:2781] T bec2e393d5e841c7857d6fd092aac53e P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:01:53.769469 31906 raft_consensus.cc:686] T bec2e393d5e841c7857d6fd092aac53e P 0db0855dad46416094c68fc878712cbb [term 1 LEADER]: Becoming Leader. State: Replica: 0db0855dad46416094c68fc878712cbb, State: Running, Role: LEADER
I20240502 02:01:53.769827 31906 consensus_queue.cc:237] T bec2e393d5e841c7857d6fd092aac53e P 0db0855dad46416094c68fc878712cbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:53.775502 31150 catalog_manager.cc:5484] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0db0855dad46416094c68fc878712cbb (127.25.109.130). New cstate: current_term: 1 leader_uuid: "0db0855dad46416094c68fc878712cbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20240502 02:01:53.780283 31150 catalog_manager.cc:5484] T bec2e393d5e841c7857d6fd092aac53e P 0db0855dad46416094c68fc878712cbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0db0855dad46416094c68fc878712cbb (127.25.109.130). New cstate: current_term: 1 leader_uuid: "0db0855dad46416094c68fc878712cbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } health_report { overall_health: UNKNOWN } } }
W20240502 02:01:53.848881 31607 tablet.cc:2369] T 28de9be2be73409387480db617b0031c P e325c0c7447f4b329994aafdee580666: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 02:01:53.863759 31474 tablet.cc:2369] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 02:01:53.875241 31340 tablet.cc:2369] T 28de9be2be73409387480db617b0031c P 64688d2615e44408ab868cb531540a84: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 02:01:54.168867 31906 consensus_queue.cc:1035] T bec2e393d5e841c7857d6fd092aac53e P 0db0855dad46416094c68fc878712cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:01:54.168788 31920 consensus_queue.cc:1035] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 02:01:54.173542 31924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:01:54.174062 31924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 02:01:54.177986 31921 consensus_queue.cc:1035] T bec2e393d5e841c7857d6fd092aac53e P 0db0855dad46416094c68fc878712cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 02:01:54.186233 31924 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 02:01:54.189514 31906 consensus_queue.cc:1035] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240502 02:01:54.248670 31907 consensus_queue.cc:1035] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:01:54.305392 31958 consensus_queue.cc:1035] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 244.83 ms
  time per row: 0.122415 ms
I20240502 02:01:54.580089 31141 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52116:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240502 02:01:54.580916 31141 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 02:01:54.589874 31270 tablet_service.cc:1467] Processing CreateTablet for tablet 171f811ca5584cd3ab08fc25a9b7f7fc (DEFAULT_TABLE table=rebalance_test_table_1 [id=1e4e557617e14b7895769013935449f4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 02:01:54.590157 31406 tablet_service.cc:1467] Processing CreateTablet for tablet 171f811ca5584cd3ab08fc25a9b7f7fc (DEFAULT_TABLE table=rebalance_test_table_1 [id=1e4e557617e14b7895769013935449f4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 02:01:54.590410 31270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 171f811ca5584cd3ab08fc25a9b7f7fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:54.590466 31537 tablet_service.cc:1467] Processing CreateTablet for tablet 171f811ca5584cd3ab08fc25a9b7f7fc (DEFAULT_TABLE table=rebalance_test_table_1 [id=1e4e557617e14b7895769013935449f4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 02:01:54.590603 31406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 171f811ca5584cd3ab08fc25a9b7f7fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:54.590879 31537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 171f811ca5584cd3ab08fc25a9b7f7fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:54.592341 31539 tablet_service.cc:1467] Processing CreateTablet for tablet a10fd1feee3a4838a8fba1599a1b890d (DEFAULT_TABLE table=rebalance_test_table_1 [id=1e4e557617e14b7895769013935449f4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 02:01:54.592751 31539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a10fd1feee3a4838a8fba1599a1b890d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:54.597293 31407 tablet_service.cc:1467] Processing CreateTablet for tablet fecea55aa2ab478d91bcad93eca93aef (DEFAULT_TABLE table=rebalance_test_table_1 [id=1e4e557617e14b7895769013935449f4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 02:01:54.597743 31407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fecea55aa2ab478d91bcad93eca93aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:54.600270 31990 tablet_bootstrap.cc:492] T 171f811ca5584cd3ab08fc25a9b7f7fc P 64688d2615e44408ab868cb531540a84: Bootstrap starting.
I20240502 02:01:54.601979 31990 tablet_bootstrap.cc:654] T 171f811ca5584cd3ab08fc25a9b7f7fc P 64688d2615e44408ab868cb531540a84: Neither blocks nor log segments found. Creating new log.
I20240502 02:01:54.603283 31989 tablet_bootstrap.cc:492] T 171f811ca5584cd3ab08fc25a9b7f7fc P 0db0855dad46416094c68fc878712cbb: Bootstrap starting.
I20240502 02:01:54.604331 31405 tablet_service.cc:1467] Processing CreateTablet for tablet a10fd1feee3a4838a8fba1599a1b890d (DEFAULT_TABLE table=rebalance_test_table_1 [id=1e4e557617e14b7895769013935449f4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 02:01:54.604321 31538 tablet_service.cc:1467] Processing CreateTablet for tablet fecea55aa2ab478d91bcad93eca93aef (DEFAULT_TABLE table=rebalance_test_table_1 [id=1e4e557617e14b7895769013935449f4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 02:01:54.604729 31538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fecea55aa2ab478d91bcad93eca93aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:54.604736 31405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a10fd1feee3a4838a8fba1599a1b890d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:54.605157 31989 tablet_bootstrap.cc:654] T 171f811ca5584cd3ab08fc25a9b7f7fc P 0db0855dad46416094c68fc878712cbb: Neither blocks nor log segments found. Creating new log.
I20240502 02:01:54.608079 31272 tablet_service.cc:1467] Processing CreateTablet for tablet fecea55aa2ab478d91bcad93eca93aef (DEFAULT_TABLE table=rebalance_test_table_1 [id=1e4e557617e14b7895769013935449f4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 02:01:54.608558 31272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fecea55aa2ab478d91bcad93eca93aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:54.612315 31271 tablet_service.cc:1467] Processing CreateTablet for tablet a10fd1feee3a4838a8fba1599a1b890d (DEFAULT_TABLE table=rebalance_test_table_1 [id=1e4e557617e14b7895769013935449f4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 02:01:54.612757 31271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a10fd1feee3a4838a8fba1599a1b890d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:54.615610 31989 tablet_bootstrap.cc:492] T 171f811ca5584cd3ab08fc25a9b7f7fc P 0db0855dad46416094c68fc878712cbb: No bootstrap required, opened a new log
I20240502 02:01:54.616499 31989 ts_tablet_manager.cc:1367] T 171f811ca5584cd3ab08fc25a9b7f7fc P 0db0855dad46416094c68fc878712cbb: Time spent bootstrapping tablet: real 0.013s	user 0.000s	sys 0.012s
I20240502 02:01:54.617642 31989 raft_consensus.cc:348] T 171f811ca5584cd3ab08fc25a9b7f7fc P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:54.617944 31989 raft_consensus.cc:374] T 171f811ca5584cd3ab08fc25a9b7f7fc P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:01:54.618060 31989 raft_consensus.cc:729] T 171f811ca5584cd3ab08fc25a9b7f7fc P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0db0855dad46416094c68fc878712cbb, State: Initialized, Role: FOLLOWER
I20240502 02:01:54.618356 31989 consensus_queue.cc:260] T 171f811ca5584cd3ab08fc25a9b7f7fc P 0db0855dad46416094c68fc878712cbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:54.619033 31989 ts_tablet_manager.cc:1398] T 171f811ca5584cd3ab08fc25a9b7f7fc P 0db0855dad46416094c68fc878712cbb: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 02:01:54.619419 31989 tablet_bootstrap.cc:492] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb: Bootstrap starting.
I20240502 02:01:54.621337 31989 tablet_bootstrap.cc:654] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb: Neither blocks nor log segments found. Creating new log.
I20240502 02:01:54.635593 31990 tablet_bootstrap.cc:492] T 171f811ca5584cd3ab08fc25a9b7f7fc P 64688d2615e44408ab868cb531540a84: No bootstrap required, opened a new log
I20240502 02:01:54.635843 31990 ts_tablet_manager.cc:1367] T 171f811ca5584cd3ab08fc25a9b7f7fc P 64688d2615e44408ab868cb531540a84: Time spent bootstrapping tablet: real 0.036s	user 0.008s	sys 0.014s
I20240502 02:01:54.637084 31990 raft_consensus.cc:348] T 171f811ca5584cd3ab08fc25a9b7f7fc P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:54.637341 31990 raft_consensus.cc:374] T 171f811ca5584cd3ab08fc25a9b7f7fc P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:01:54.637472 31990 raft_consensus.cc:729] T 171f811ca5584cd3ab08fc25a9b7f7fc P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64688d2615e44408ab868cb531540a84, State: Initialized, Role: FOLLOWER
I20240502 02:01:54.637794 31990 consensus_queue.cc:260] T 171f811ca5584cd3ab08fc25a9b7f7fc P 64688d2615e44408ab868cb531540a84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:54.640172 31990 ts_tablet_manager.cc:1398] T 171f811ca5584cd3ab08fc25a9b7f7fc P 64688d2615e44408ab868cb531540a84: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 02:01:54.642925 31990 tablet_bootstrap.cc:492] T a10fd1feee3a4838a8fba1599a1b890d P 64688d2615e44408ab868cb531540a84: Bootstrap starting.
I20240502 02:01:54.645998 31990 tablet_bootstrap.cc:654] T a10fd1feee3a4838a8fba1599a1b890d P 64688d2615e44408ab868cb531540a84: Neither blocks nor log segments found. Creating new log.
I20240502 02:01:54.652485 31991 tablet_bootstrap.cc:492] T a10fd1feee3a4838a8fba1599a1b890d P e325c0c7447f4b329994aafdee580666: Bootstrap starting.
I20240502 02:01:54.654587 31991 tablet_bootstrap.cc:654] T a10fd1feee3a4838a8fba1599a1b890d P e325c0c7447f4b329994aafdee580666: Neither blocks nor log segments found. Creating new log.
I20240502 02:01:54.660444 31990 tablet_bootstrap.cc:492] T a10fd1feee3a4838a8fba1599a1b890d P 64688d2615e44408ab868cb531540a84: No bootstrap required, opened a new log
I20240502 02:01:54.660682 31990 ts_tablet_manager.cc:1367] T a10fd1feee3a4838a8fba1599a1b890d P 64688d2615e44408ab868cb531540a84: Time spent bootstrapping tablet: real 0.020s	user 0.004s	sys 0.000s
I20240502 02:01:54.661603 31990 raft_consensus.cc:348] T a10fd1feee3a4838a8fba1599a1b890d P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:54.661890 31990 raft_consensus.cc:374] T a10fd1feee3a4838a8fba1599a1b890d P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:01:54.661973 31990 raft_consensus.cc:729] T a10fd1feee3a4838a8fba1599a1b890d P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64688d2615e44408ab868cb531540a84, State: Initialized, Role: FOLLOWER
I20240502 02:01:54.662264 31990 consensus_queue.cc:260] T a10fd1feee3a4838a8fba1599a1b890d P 64688d2615e44408ab868cb531540a84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:54.662992 31928 raft_consensus.cc:482] T 171f811ca5584cd3ab08fc25a9b7f7fc P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 02:01:54.663167 31928 raft_consensus.cc:504] T 171f811ca5584cd3ab08fc25a9b7f7fc P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:54.665278 31928 leader_election.cc:288] T 171f811ca5584cd3ab08fc25a9b7f7fc P 64688d2615e44408ab868cb531540a84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0db0855dad46416094c68fc878712cbb (127.25.109.130:38025), e325c0c7447f4b329994aafdee580666 (127.25.109.131:43937)
I20240502 02:01:54.665441 31991 tablet_bootstrap.cc:492] T a10fd1feee3a4838a8fba1599a1b890d P e325c0c7447f4b329994aafdee580666: No bootstrap required, opened a new log
I20240502 02:01:54.665603 31991 ts_tablet_manager.cc:1367] T a10fd1feee3a4838a8fba1599a1b890d P e325c0c7447f4b329994aafdee580666: Time spent bootstrapping tablet: real 0.013s	user 0.009s	sys 0.000s
I20240502 02:01:54.666525 31991 raft_consensus.cc:348] T a10fd1feee3a4838a8fba1599a1b890d P e325c0c7447f4b329994aafdee580666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:54.666755 31991 raft_consensus.cc:374] T a10fd1feee3a4838a8fba1599a1b890d P e325c0c7447f4b329994aafdee580666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:01:54.666849 31991 raft_consensus.cc:729] T a10fd1feee3a4838a8fba1599a1b890d P e325c0c7447f4b329994aafdee580666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e325c0c7447f4b329994aafdee580666, State: Initialized, Role: FOLLOWER
I20240502 02:01:54.667385 31990 ts_tablet_manager.cc:1398] T a10fd1feee3a4838a8fba1599a1b890d P 64688d2615e44408ab868cb531540a84: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.000s
I20240502 02:01:54.667138 31991 consensus_queue.cc:260] T a10fd1feee3a4838a8fba1599a1b890d P e325c0c7447f4b329994aafdee580666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:54.668149 31991 ts_tablet_manager.cc:1398] T a10fd1feee3a4838a8fba1599a1b890d P e325c0c7447f4b329994aafdee580666: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 02:01:54.668511 31990 tablet_bootstrap.cc:492] T fecea55aa2ab478d91bcad93eca93aef P 64688d2615e44408ab868cb531540a84: Bootstrap starting.
I20240502 02:01:54.668524 31991 tablet_bootstrap.cc:492] T 171f811ca5584cd3ab08fc25a9b7f7fc P e325c0c7447f4b329994aafdee580666: Bootstrap starting.
I20240502 02:01:54.670318 31991 tablet_bootstrap.cc:654] T 171f811ca5584cd3ab08fc25a9b7f7fc P e325c0c7447f4b329994aafdee580666: Neither blocks nor log segments found. Creating new log.
I20240502 02:01:54.673483 31990 tablet_bootstrap.cc:654] T fecea55aa2ab478d91bcad93eca93aef P 64688d2615e44408ab868cb531540a84: Neither blocks nor log segments found. Creating new log.
I20240502 02:01:54.684315 31989 tablet_bootstrap.cc:492] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb: No bootstrap required, opened a new log
I20240502 02:01:54.684643 31989 ts_tablet_manager.cc:1367] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb: Time spent bootstrapping tablet: real 0.065s	user 0.030s	sys 0.031s
I20240502 02:01:54.685601 31989 raft_consensus.cc:348] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:54.685905 31989 raft_consensus.cc:374] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:01:54.686005 31989 raft_consensus.cc:729] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0db0855dad46416094c68fc878712cbb, State: Initialized, Role: FOLLOWER
I20240502 02:01:54.686350 31989 consensus_queue.cc:260] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:54.686988 31989 ts_tablet_manager.cc:1398] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 02:01:54.687369 31989 tablet_bootstrap.cc:492] T a10fd1feee3a4838a8fba1599a1b890d P 0db0855dad46416094c68fc878712cbb: Bootstrap starting.
I20240502 02:01:54.689148 31989 tablet_bootstrap.cc:654] T a10fd1feee3a4838a8fba1599a1b890d P 0db0855dad46416094c68fc878712cbb: Neither blocks nor log segments found. Creating new log.
I20240502 02:01:54.716008 31990 tablet_bootstrap.cc:492] T fecea55aa2ab478d91bcad93eca93aef P 64688d2615e44408ab868cb531540a84: No bootstrap required, opened a new log
I20240502 02:01:54.716362 31990 ts_tablet_manager.cc:1367] T fecea55aa2ab478d91bcad93eca93aef P 64688d2615e44408ab868cb531540a84: Time spent bootstrapping tablet: real 0.049s	user 0.009s	sys 0.011s
I20240502 02:01:54.717617 31990 raft_consensus.cc:348] T fecea55aa2ab478d91bcad93eca93aef P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:54.718024 31990 raft_consensus.cc:374] T fecea55aa2ab478d91bcad93eca93aef P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:01:54.718173 31990 raft_consensus.cc:729] T fecea55aa2ab478d91bcad93eca93aef P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64688d2615e44408ab868cb531540a84, State: Initialized, Role: FOLLOWER
I20240502 02:01:54.718693 31990 consensus_queue.cc:260] T fecea55aa2ab478d91bcad93eca93aef P 64688d2615e44408ab868cb531540a84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:54.719622 31990 ts_tablet_manager.cc:1398] T fecea55aa2ab478d91bcad93eca93aef P 64688d2615e44408ab868cb531540a84: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 02:01:54.723021 31989 tablet_bootstrap.cc:492] T a10fd1feee3a4838a8fba1599a1b890d P 0db0855dad46416094c68fc878712cbb: No bootstrap required, opened a new log
I20240502 02:01:54.723244 31989 ts_tablet_manager.cc:1367] T a10fd1feee3a4838a8fba1599a1b890d P 0db0855dad46416094c68fc878712cbb: Time spent bootstrapping tablet: real 0.036s	user 0.015s	sys 0.018s
I20240502 02:01:54.724265 31991 tablet_bootstrap.cc:492] T 171f811ca5584cd3ab08fc25a9b7f7fc P e325c0c7447f4b329994aafdee580666: No bootstrap required, opened a new log
I20240502 02:01:54.724175 31428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "171f811ca5584cd3ab08fc25a9b7f7fc" candidate_uuid: "64688d2615e44408ab868cb531540a84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0db0855dad46416094c68fc878712cbb" is_pre_election: true
I20240502 02:01:54.724288 31989 raft_consensus.cc:348] T a10fd1feee3a4838a8fba1599a1b890d P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:54.724448 31428 raft_consensus.cc:2454] T 171f811ca5584cd3ab08fc25a9b7f7fc P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64688d2615e44408ab868cb531540a84 in term 0.
I20240502 02:01:54.724514 31989 raft_consensus.cc:374] T a10fd1feee3a4838a8fba1599a1b890d P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:01:54.724604 31989 raft_consensus.cc:729] T a10fd1feee3a4838a8fba1599a1b890d P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0db0855dad46416094c68fc878712cbb, State: Initialized, Role: FOLLOWER
I20240502 02:01:54.724797 31991 ts_tablet_manager.cc:1367] T 171f811ca5584cd3ab08fc25a9b7f7fc P e325c0c7447f4b329994aafdee580666: Time spent bootstrapping tablet: real 0.056s	user 0.029s	sys 0.025s
I20240502 02:01:54.724890 31989 consensus_queue.cc:260] T a10fd1feee3a4838a8fba1599a1b890d P 0db0855dad46416094c68fc878712cbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:54.724975 31558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "171f811ca5584cd3ab08fc25a9b7f7fc" candidate_uuid: "64688d2615e44408ab868cb531540a84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e325c0c7447f4b329994aafdee580666" is_pre_election: true
I20240502 02:01:54.725173 31227 leader_election.cc:302] T 171f811ca5584cd3ab08fc25a9b7f7fc P 64688d2615e44408ab868cb531540a84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0db0855dad46416094c68fc878712cbb, 64688d2615e44408ab868cb531540a84; no voters: 
I20240502 02:01:54.725615 31928 raft_consensus.cc:2781] T 171f811ca5584cd3ab08fc25a9b7f7fc P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 02:01:54.725754 31928 raft_consensus.cc:482] T 171f811ca5584cd3ab08fc25a9b7f7fc P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 02:01:54.725858 31928 raft_consensus.cc:3037] T 171f811ca5584cd3ab08fc25a9b7f7fc P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:01:54.726140 31991 raft_consensus.cc:348] T 171f811ca5584cd3ab08fc25a9b7f7fc P e325c0c7447f4b329994aafdee580666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:54.726459 31991 raft_consensus.cc:374] T 171f811ca5584cd3ab08fc25a9b7f7fc P e325c0c7447f4b329994aafdee580666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:01:54.726593 31991 raft_consensus.cc:729] T 171f811ca5584cd3ab08fc25a9b7f7fc P e325c0c7447f4b329994aafdee580666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e325c0c7447f4b329994aafdee580666, State: Initialized, Role: FOLLOWER
I20240502 02:01:54.726930 31991 consensus_queue.cc:260] T 171f811ca5584cd3ab08fc25a9b7f7fc P e325c0c7447f4b329994aafdee580666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:54.727530 31991 ts_tablet_manager.cc:1398] T 171f811ca5584cd3ab08fc25a9b7f7fc P e325c0c7447f4b329994aafdee580666: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 02:01:54.727732 31989 ts_tablet_manager.cc:1398] T a10fd1feee3a4838a8fba1599a1b890d P 0db0855dad46416094c68fc878712cbb: Time spent starting tablet: real 0.004s	user 0.003s	sys 0.000s
I20240502 02:01:54.728031 31991 tablet_bootstrap.cc:492] T fecea55aa2ab478d91bcad93eca93aef P e325c0c7447f4b329994aafdee580666: Bootstrap starting.
W20240502 02:01:54.728523 31226 leader_election.cc:341] T 171f811ca5584cd3ab08fc25a9b7f7fc P 64688d2615e44408ab868cb531540a84 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e325c0c7447f4b329994aafdee580666 (127.25.109.131:43937): Illegal state: must be running to vote when last-logged opid is not known
I20240502 02:01:54.728821 31928 raft_consensus.cc:504] T 171f811ca5584cd3ab08fc25a9b7f7fc P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:54.729460 31991 tablet_bootstrap.cc:654] T fecea55aa2ab478d91bcad93eca93aef P e325c0c7447f4b329994aafdee580666: Neither blocks nor log segments found. Creating new log.
I20240502 02:01:54.729557 31928 leader_election.cc:288] T 171f811ca5584cd3ab08fc25a9b7f7fc P 64688d2615e44408ab868cb531540a84 [CANDIDATE]: Term 1 election: Requested vote from peers 0db0855dad46416094c68fc878712cbb (127.25.109.130:38025), e325c0c7447f4b329994aafdee580666 (127.25.109.131:43937)
I20240502 02:01:54.730232 31558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "171f811ca5584cd3ab08fc25a9b7f7fc" candidate_uuid: "64688d2615e44408ab868cb531540a84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e325c0c7447f4b329994aafdee580666"
I20240502 02:01:54.730468 31558 raft_consensus.cc:3037] T 171f811ca5584cd3ab08fc25a9b7f7fc P e325c0c7447f4b329994aafdee580666 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:01:54.731850 31428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "171f811ca5584cd3ab08fc25a9b7f7fc" candidate_uuid: "64688d2615e44408ab868cb531540a84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0db0855dad46416094c68fc878712cbb"
I20240502 02:01:54.732128 31428 raft_consensus.cc:3037] T 171f811ca5584cd3ab08fc25a9b7f7fc P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:01:54.733652 31558 raft_consensus.cc:2454] T 171f811ca5584cd3ab08fc25a9b7f7fc P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64688d2615e44408ab868cb531540a84 in term 1.
I20240502 02:01:54.734161 31226 leader_election.cc:302] T 171f811ca5584cd3ab08fc25a9b7f7fc P 64688d2615e44408ab868cb531540a84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64688d2615e44408ab868cb531540a84, e325c0c7447f4b329994aafdee580666; no voters: 
I20240502 02:01:54.734481 31928 raft_consensus.cc:2781] T 171f811ca5584cd3ab08fc25a9b7f7fc P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:01:54.735071 31991 tablet_bootstrap.cc:492] T fecea55aa2ab478d91bcad93eca93aef P e325c0c7447f4b329994aafdee580666: No bootstrap required, opened a new log
I20240502 02:01:54.735107 31428 raft_consensus.cc:2454] T 171f811ca5584cd3ab08fc25a9b7f7fc P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64688d2615e44408ab868cb531540a84 in term 1.
I20240502 02:01:54.735567 31991 ts_tablet_manager.cc:1367] T fecea55aa2ab478d91bcad93eca93aef P e325c0c7447f4b329994aafdee580666: Time spent bootstrapping tablet: real 0.007s	user 0.007s	sys 0.000s
I20240502 02:01:54.735631 31928 raft_consensus.cc:686] T 171f811ca5584cd3ab08fc25a9b7f7fc P 64688d2615e44408ab868cb531540a84 [term 1 LEADER]: Becoming Leader. State: Replica: 64688d2615e44408ab868cb531540a84, State: Running, Role: LEADER
I20240502 02:01:54.736208 31928 consensus_queue.cc:237] T 171f811ca5584cd3ab08fc25a9b7f7fc P 64688d2615e44408ab868cb531540a84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:54.736399 31991 raft_consensus.cc:348] T fecea55aa2ab478d91bcad93eca93aef P e325c0c7447f4b329994aafdee580666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:54.736595 31991 raft_consensus.cc:374] T fecea55aa2ab478d91bcad93eca93aef P e325c0c7447f4b329994aafdee580666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:01:54.736671 31991 raft_consensus.cc:729] T fecea55aa2ab478d91bcad93eca93aef P e325c0c7447f4b329994aafdee580666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e325c0c7447f4b329994aafdee580666, State: Initialized, Role: FOLLOWER
I20240502 02:01:54.736917 31991 consensus_queue.cc:260] T fecea55aa2ab478d91bcad93eca93aef P e325c0c7447f4b329994aafdee580666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:54.737429 31991 ts_tablet_manager.cc:1398] T fecea55aa2ab478d91bcad93eca93aef P e325c0c7447f4b329994aafdee580666: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 02:01:54.739899 31141 catalog_manager.cc:5484] T 171f811ca5584cd3ab08fc25a9b7f7fc P 64688d2615e44408ab868cb531540a84 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64688d2615e44408ab868cb531540a84 (127.25.109.129). New cstate: current_term: 1 leader_uuid: "64688d2615e44408ab868cb531540a84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20240502 02:01:54.960322 31928 raft_consensus.cc:482] T a10fd1feee3a4838a8fba1599a1b890d P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 02:01:54.960556 31928 raft_consensus.cc:504] T a10fd1feee3a4838a8fba1599a1b890d P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:54.961210 31928 leader_election.cc:288] T a10fd1feee3a4838a8fba1599a1b890d P 64688d2615e44408ab868cb531540a84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0db0855dad46416094c68fc878712cbb (127.25.109.130:38025), e325c0c7447f4b329994aafdee580666 (127.25.109.131:43937)
I20240502 02:01:54.961850 31428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a10fd1feee3a4838a8fba1599a1b890d" candidate_uuid: "64688d2615e44408ab868cb531540a84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0db0855dad46416094c68fc878712cbb" is_pre_election: true
I20240502 02:01:54.961885 31558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a10fd1feee3a4838a8fba1599a1b890d" candidate_uuid: "64688d2615e44408ab868cb531540a84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e325c0c7447f4b329994aafdee580666" is_pre_election: true
I20240502 02:01:54.962101 31428 raft_consensus.cc:2454] T a10fd1feee3a4838a8fba1599a1b890d P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64688d2615e44408ab868cb531540a84 in term 0.
I20240502 02:01:54.962128 31558 raft_consensus.cc:2454] T a10fd1feee3a4838a8fba1599a1b890d P e325c0c7447f4b329994aafdee580666 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64688d2615e44408ab868cb531540a84 in term 0.
I20240502 02:01:54.962621 31227 leader_election.cc:302] T a10fd1feee3a4838a8fba1599a1b890d P 64688d2615e44408ab868cb531540a84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0db0855dad46416094c68fc878712cbb, 64688d2615e44408ab868cb531540a84; no voters: 
I20240502 02:01:54.962944 31928 raft_consensus.cc:2781] T a10fd1feee3a4838a8fba1599a1b890d P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 02:01:54.963037 31928 raft_consensus.cc:482] T a10fd1feee3a4838a8fba1599a1b890d P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 02:01:54.963112 31928 raft_consensus.cc:3037] T a10fd1feee3a4838a8fba1599a1b890d P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:01:54.965085 31928 raft_consensus.cc:504] T a10fd1feee3a4838a8fba1599a1b890d P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:54.965703 31928 leader_election.cc:288] T a10fd1feee3a4838a8fba1599a1b890d P 64688d2615e44408ab868cb531540a84 [CANDIDATE]: Term 1 election: Requested vote from peers 0db0855dad46416094c68fc878712cbb (127.25.109.130:38025), e325c0c7447f4b329994aafdee580666 (127.25.109.131:43937)
I20240502 02:01:54.966241 31558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a10fd1feee3a4838a8fba1599a1b890d" candidate_uuid: "64688d2615e44408ab868cb531540a84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e325c0c7447f4b329994aafdee580666"
I20240502 02:01:54.966225 31428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a10fd1feee3a4838a8fba1599a1b890d" candidate_uuid: "64688d2615e44408ab868cb531540a84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0db0855dad46416094c68fc878712cbb"
I20240502 02:01:54.966464 31558 raft_consensus.cc:3037] T a10fd1feee3a4838a8fba1599a1b890d P e325c0c7447f4b329994aafdee580666 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:01:54.966485 31428 raft_consensus.cc:3037] T a10fd1feee3a4838a8fba1599a1b890d P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:01:54.969221 31428 raft_consensus.cc:2454] T a10fd1feee3a4838a8fba1599a1b890d P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64688d2615e44408ab868cb531540a84 in term 1.
I20240502 02:01:54.969247 31558 raft_consensus.cc:2454] T a10fd1feee3a4838a8fba1599a1b890d P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64688d2615e44408ab868cb531540a84 in term 1.
I20240502 02:01:54.969658 31227 leader_election.cc:302] T a10fd1feee3a4838a8fba1599a1b890d P 64688d2615e44408ab868cb531540a84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0db0855dad46416094c68fc878712cbb, 64688d2615e44408ab868cb531540a84; no voters: 
I20240502 02:01:54.969952 31928 raft_consensus.cc:2781] T a10fd1feee3a4838a8fba1599a1b890d P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:01:54.970129 31928 raft_consensus.cc:686] T a10fd1feee3a4838a8fba1599a1b890d P 64688d2615e44408ab868cb531540a84 [term 1 LEADER]: Becoming Leader. State: Replica: 64688d2615e44408ab868cb531540a84, State: Running, Role: LEADER
I20240502 02:01:54.970410 31928 consensus_queue.cc:237] T a10fd1feee3a4838a8fba1599a1b890d P 64688d2615e44408ab868cb531540a84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:54.970949 31921 raft_consensus.cc:482] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 02:01:54.971104 31921 raft_consensus.cc:504] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:54.971786 31921 leader_election.cc:288] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64688d2615e44408ab868cb531540a84 (127.25.109.129:33021), e325c0c7447f4b329994aafdee580666 (127.25.109.131:43937)
I20240502 02:01:54.972467 31558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fecea55aa2ab478d91bcad93eca93aef" candidate_uuid: "0db0855dad46416094c68fc878712cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e325c0c7447f4b329994aafdee580666" is_pre_election: true
I20240502 02:01:54.972754 31558 raft_consensus.cc:2454] T fecea55aa2ab478d91bcad93eca93aef P e325c0c7447f4b329994aafdee580666 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0db0855dad46416094c68fc878712cbb in term 0.
I20240502 02:01:54.972808 31292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fecea55aa2ab478d91bcad93eca93aef" candidate_uuid: "0db0855dad46416094c68fc878712cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64688d2615e44408ab868cb531540a84" is_pre_election: true
I20240502 02:01:54.973006 31292 raft_consensus.cc:2454] T fecea55aa2ab478d91bcad93eca93aef P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0db0855dad46416094c68fc878712cbb in term 0.
I20240502 02:01:54.973174 31361 leader_election.cc:302] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0db0855dad46416094c68fc878712cbb, e325c0c7447f4b329994aafdee580666; no voters: 
I20240502 02:01:54.973495 31921 raft_consensus.cc:2781] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 02:01:54.973578 31921 raft_consensus.cc:482] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 02:01:54.973649 31921 raft_consensus.cc:3037] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:01:54.973487 31141 catalog_manager.cc:5484] T a10fd1feee3a4838a8fba1599a1b890d P 64688d2615e44408ab868cb531540a84 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64688d2615e44408ab868cb531540a84 (127.25.109.129). New cstate: current_term: 1 leader_uuid: "64688d2615e44408ab868cb531540a84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20240502 02:01:54.975483 31921 raft_consensus.cc:504] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:54.976182 31921 leader_election.cc:288] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb [CANDIDATE]: Term 1 election: Requested vote from peers 64688d2615e44408ab868cb531540a84 (127.25.109.129:33021), e325c0c7447f4b329994aafdee580666 (127.25.109.131:43937)
I20240502 02:01:54.976805 31292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fecea55aa2ab478d91bcad93eca93aef" candidate_uuid: "0db0855dad46416094c68fc878712cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64688d2615e44408ab868cb531540a84"
I20240502 02:01:54.977030 31292 raft_consensus.cc:3037] T fecea55aa2ab478d91bcad93eca93aef P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:01:54.976922 31558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fecea55aa2ab478d91bcad93eca93aef" candidate_uuid: "0db0855dad46416094c68fc878712cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e325c0c7447f4b329994aafdee580666"
I20240502 02:01:54.977242 31558 raft_consensus.cc:3037] T fecea55aa2ab478d91bcad93eca93aef P e325c0c7447f4b329994aafdee580666 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:01:54.979116 31558 raft_consensus.cc:2454] T fecea55aa2ab478d91bcad93eca93aef P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0db0855dad46416094c68fc878712cbb in term 1.
I20240502 02:01:54.979131 31292 raft_consensus.cc:2454] T fecea55aa2ab478d91bcad93eca93aef P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0db0855dad46416094c68fc878712cbb in term 1.
I20240502 02:01:54.979487 31361 leader_election.cc:302] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0db0855dad46416094c68fc878712cbb, 64688d2615e44408ab868cb531540a84; no voters: 
I20240502 02:01:54.979771 31921 raft_consensus.cc:2781] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:01:54.980219 31921 raft_consensus.cc:686] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb [term 1 LEADER]: Becoming Leader. State: Replica: 0db0855dad46416094c68fc878712cbb, State: Running, Role: LEADER
I20240502 02:01:54.980545 31921 consensus_queue.cc:237] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:54.983233 31141 catalog_manager.cc:5484] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0db0855dad46416094c68fc878712cbb (127.25.109.130). New cstate: current_term: 1 leader_uuid: "0db0855dad46416094c68fc878712cbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20240502 02:01:55.333876 31928 consensus_queue.cc:1035] T 171f811ca5584cd3ab08fc25a9b7f7fc P 64688d2615e44408ab868cb531540a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:01:55.343564 32003 consensus_queue.cc:1035] T 171f811ca5584cd3ab08fc25a9b7f7fc P 64688d2615e44408ab868cb531540a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:01:55.402623 32003 consensus_queue.cc:1035] T a10fd1feee3a4838a8fba1599a1b890d P 64688d2615e44408ab868cb531540a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:01:55.408162 32003 consensus_queue.cc:1035] T a10fd1feee3a4838a8fba1599a1b890d P 64688d2615e44408ab868cb531540a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 02:01:55.426337 32008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:01:55.426721 32008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:01:55.434482 32008 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 02:01:55.550806 31921 consensus_queue.cc:1035] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:01:55.589255 31559 raft_consensus.cc:1261] T fecea55aa2ab478d91bcad93eca93aef P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Refusing update from remote peer 0db0855dad46416094c68fc878712cbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 02:01:55.591431 31921 consensus_queue.cc:1035] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:01:55.671061 32056 mvcc.cc:205] Tried to move back new op lower bound from 7023064332595867648 to 7023064330163146752. Current Snapshot: MvccSnapshot[applied={T|T < 7023064332595867648}]

INSERT report
    rows total: 2000
    time total: 223.529 ms
  time per row: 0.111765 ms
I20240502 02:01:55.808373 31147 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52116:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240502 02:01:55.809167 31147 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 02:01:55.818441 31270 tablet_service.cc:1467] Processing CreateTablet for tablet 1b97531b47d44bf8a7f905ef339f857a (DEFAULT_TABLE table=rebalance_test_table_2 [id=a1611faa4bb54872a21542e38fdc1e77]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 02:01:55.818583 31407 tablet_service.cc:1467] Processing CreateTablet for tablet 1b97531b47d44bf8a7f905ef339f857a (DEFAULT_TABLE table=rebalance_test_table_2 [id=a1611faa4bb54872a21542e38fdc1e77]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 02:01:55.819021 31407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b97531b47d44bf8a7f905ef339f857a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:55.819025 31270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b97531b47d44bf8a7f905ef339f857a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:55.820283 31539 tablet_service.cc:1467] Processing CreateTablet for tablet 1b97531b47d44bf8a7f905ef339f857a (DEFAULT_TABLE table=rebalance_test_table_2 [id=a1611faa4bb54872a21542e38fdc1e77]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 02:01:55.820288 31272 tablet_service.cc:1467] Processing CreateTablet for tablet 37456bdef9f94a52b63d0ab75f8ffa8b (DEFAULT_TABLE table=rebalance_test_table_2 [id=a1611faa4bb54872a21542e38fdc1e77]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 02:01:55.820348 31537 tablet_service.cc:1467] Processing CreateTablet for tablet 9676b44f15e84ae28a64caa71f5722c6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a1611faa4bb54872a21542e38fdc1e77]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 02:01:55.820708 31539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b97531b47d44bf8a7f905ef339f857a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:55.820726 31272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37456bdef9f94a52b63d0ab75f8ffa8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:55.824234 31537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9676b44f15e84ae28a64caa71f5722c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:55.824407 31538 tablet_service.cc:1467] Processing CreateTablet for tablet 37456bdef9f94a52b63d0ab75f8ffa8b (DEFAULT_TABLE table=rebalance_test_table_2 [id=a1611faa4bb54872a21542e38fdc1e77]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 02:01:55.824815 31538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37456bdef9f94a52b63d0ab75f8ffa8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:55.826368 31405 tablet_service.cc:1467] Processing CreateTablet for tablet 37456bdef9f94a52b63d0ab75f8ffa8b (DEFAULT_TABLE table=rebalance_test_table_2 [id=a1611faa4bb54872a21542e38fdc1e77]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 02:01:55.826821 31405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37456bdef9f94a52b63d0ab75f8ffa8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:55.826939 31271 tablet_service.cc:1467] Processing CreateTablet for tablet 9676b44f15e84ae28a64caa71f5722c6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a1611faa4bb54872a21542e38fdc1e77]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 02:01:55.827428 31271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9676b44f15e84ae28a64caa71f5722c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:55.827538 31406 tablet_service.cc:1467] Processing CreateTablet for tablet 9676b44f15e84ae28a64caa71f5722c6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=a1611faa4bb54872a21542e38fdc1e77]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 02:01:55.828013 31406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9676b44f15e84ae28a64caa71f5722c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:55.837185 32069 tablet_bootstrap.cc:492] T 1b97531b47d44bf8a7f905ef339f857a P 64688d2615e44408ab868cb531540a84: Bootstrap starting.
I20240502 02:01:55.838917 32069 tablet_bootstrap.cc:654] T 1b97531b47d44bf8a7f905ef339f857a P 64688d2615e44408ab868cb531540a84: Neither blocks nor log segments found. Creating new log.
I20240502 02:01:55.847777 32072 tablet_bootstrap.cc:492] T 9676b44f15e84ae28a64caa71f5722c6 P e325c0c7447f4b329994aafdee580666: Bootstrap starting.
I20240502 02:01:55.849288 32069 tablet_bootstrap.cc:492] T 1b97531b47d44bf8a7f905ef339f857a P 64688d2615e44408ab868cb531540a84: No bootstrap required, opened a new log
I20240502 02:01:55.849479 32069 ts_tablet_manager.cc:1367] T 1b97531b47d44bf8a7f905ef339f857a P 64688d2615e44408ab868cb531540a84: Time spent bootstrapping tablet: real 0.012s	user 0.011s	sys 0.000s
I20240502 02:01:55.849742 32072 tablet_bootstrap.cc:654] T 9676b44f15e84ae28a64caa71f5722c6 P e325c0c7447f4b329994aafdee580666: Neither blocks nor log segments found. Creating new log.
I20240502 02:01:55.850410 32069 raft_consensus.cc:348] T 1b97531b47d44bf8a7f905ef339f857a P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:55.850641 32069 raft_consensus.cc:374] T 1b97531b47d44bf8a7f905ef339f857a P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:01:55.850747 32069 raft_consensus.cc:729] T 1b97531b47d44bf8a7f905ef339f857a P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64688d2615e44408ab868cb531540a84, State: Initialized, Role: FOLLOWER
I20240502 02:01:55.851059 32070 tablet_bootstrap.cc:492] T 1b97531b47d44bf8a7f905ef339f857a P 0db0855dad46416094c68fc878712cbb: Bootstrap starting.
I20240502 02:01:55.851034 32069 consensus_queue.cc:260] T 1b97531b47d44bf8a7f905ef339f857a P 64688d2615e44408ab868cb531540a84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:55.851694 32069 ts_tablet_manager.cc:1398] T 1b97531b47d44bf8a7f905ef339f857a P 64688d2615e44408ab868cb531540a84: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 02:01:55.852043 32069 tablet_bootstrap.cc:492] T 9676b44f15e84ae28a64caa71f5722c6 P 64688d2615e44408ab868cb531540a84: Bootstrap starting.
I20240502 02:01:55.852960 32070 tablet_bootstrap.cc:654] T 1b97531b47d44bf8a7f905ef339f857a P 0db0855dad46416094c68fc878712cbb: Neither blocks nor log segments found. Creating new log.
I20240502 02:01:55.853758 32069 tablet_bootstrap.cc:654] T 9676b44f15e84ae28a64caa71f5722c6 P 64688d2615e44408ab868cb531540a84: Neither blocks nor log segments found. Creating new log.
I20240502 02:01:55.858770 32072 tablet_bootstrap.cc:492] T 9676b44f15e84ae28a64caa71f5722c6 P e325c0c7447f4b329994aafdee580666: No bootstrap required, opened a new log
I20240502 02:01:55.858949 32072 ts_tablet_manager.cc:1367] T 9676b44f15e84ae28a64caa71f5722c6 P e325c0c7447f4b329994aafdee580666: Time spent bootstrapping tablet: real 0.011s	user 0.000s	sys 0.009s
I20240502 02:01:55.859557 32070 tablet_bootstrap.cc:492] T 1b97531b47d44bf8a7f905ef339f857a P 0db0855dad46416094c68fc878712cbb: No bootstrap required, opened a new log
I20240502 02:01:55.859604 32069 tablet_bootstrap.cc:492] T 9676b44f15e84ae28a64caa71f5722c6 P 64688d2615e44408ab868cb531540a84: No bootstrap required, opened a new log
I20240502 02:01:55.859783 32070 ts_tablet_manager.cc:1367] T 1b97531b47d44bf8a7f905ef339f857a P 0db0855dad46416094c68fc878712cbb: Time spent bootstrapping tablet: real 0.009s	user 0.004s	sys 0.004s
I20240502 02:01:55.859846 32069 ts_tablet_manager.cc:1367] T 9676b44f15e84ae28a64caa71f5722c6 P 64688d2615e44408ab868cb531540a84: Time spent bootstrapping tablet: real 0.008s	user 0.001s	sys 0.005s
I20240502 02:01:55.859849 32072 raft_consensus.cc:348] T 9676b44f15e84ae28a64caa71f5722c6 P e325c0c7447f4b329994aafdee580666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:55.860116 32072 raft_consensus.cc:374] T 9676b44f15e84ae28a64caa71f5722c6 P e325c0c7447f4b329994aafdee580666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:01:55.860213 32072 raft_consensus.cc:729] T 9676b44f15e84ae28a64caa71f5722c6 P e325c0c7447f4b329994aafdee580666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e325c0c7447f4b329994aafdee580666, State: Initialized, Role: FOLLOWER
I20240502 02:01:55.860471 32072 consensus_queue.cc:260] T 9676b44f15e84ae28a64caa71f5722c6 P e325c0c7447f4b329994aafdee580666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:55.860618 32070 raft_consensus.cc:348] T 1b97531b47d44bf8a7f905ef339f857a P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:55.860827 32070 raft_consensus.cc:374] T 1b97531b47d44bf8a7f905ef339f857a P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:01:55.860743 32069 raft_consensus.cc:348] T 9676b44f15e84ae28a64caa71f5722c6 P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:55.860921 32070 raft_consensus.cc:729] T 1b97531b47d44bf8a7f905ef339f857a P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0db0855dad46416094c68fc878712cbb, State: Initialized, Role: FOLLOWER
I20240502 02:01:55.860967 32069 raft_consensus.cc:374] T 9676b44f15e84ae28a64caa71f5722c6 P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:01:55.861043 32069 raft_consensus.cc:729] T 9676b44f15e84ae28a64caa71f5722c6 P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64688d2615e44408ab868cb531540a84, State: Initialized, Role: FOLLOWER
I20240502 02:01:55.861124 32072 ts_tablet_manager.cc:1398] T 9676b44f15e84ae28a64caa71f5722c6 P e325c0c7447f4b329994aafdee580666: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 02:01:55.861198 32070 consensus_queue.cc:260] T 1b97531b47d44bf8a7f905ef339f857a P 0db0855dad46416094c68fc878712cbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:55.861305 32069 consensus_queue.cc:260] T 9676b44f15e84ae28a64caa71f5722c6 P 64688d2615e44408ab868cb531540a84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:55.861433 32072 tablet_bootstrap.cc:492] T 1b97531b47d44bf8a7f905ef339f857a P e325c0c7447f4b329994aafdee580666: Bootstrap starting.
I20240502 02:01:55.862112 32069 ts_tablet_manager.cc:1398] T 9676b44f15e84ae28a64caa71f5722c6 P 64688d2615e44408ab868cb531540a84: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 02:01:55.862483 32069 tablet_bootstrap.cc:492] T 37456bdef9f94a52b63d0ab75f8ffa8b P 64688d2615e44408ab868cb531540a84: Bootstrap starting.
I20240502 02:01:55.862577 32070 ts_tablet_manager.cc:1398] T 1b97531b47d44bf8a7f905ef339f857a P 0db0855dad46416094c68fc878712cbb: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 02:01:55.862885 32070 tablet_bootstrap.cc:492] T 9676b44f15e84ae28a64caa71f5722c6 P 0db0855dad46416094c68fc878712cbb: Bootstrap starting.
I20240502 02:01:55.863238 32072 tablet_bootstrap.cc:654] T 1b97531b47d44bf8a7f905ef339f857a P e325c0c7447f4b329994aafdee580666: Neither blocks nor log segments found. Creating new log.
I20240502 02:01:55.864218 32069 tablet_bootstrap.cc:654] T 37456bdef9f94a52b63d0ab75f8ffa8b P 64688d2615e44408ab868cb531540a84: Neither blocks nor log segments found. Creating new log.
I20240502 02:01:55.864571 32070 tablet_bootstrap.cc:654] T 9676b44f15e84ae28a64caa71f5722c6 P 0db0855dad46416094c68fc878712cbb: Neither blocks nor log segments found. Creating new log.
I20240502 02:01:55.873760 32069 tablet_bootstrap.cc:492] T 37456bdef9f94a52b63d0ab75f8ffa8b P 64688d2615e44408ab868cb531540a84: No bootstrap required, opened a new log
I20240502 02:01:55.873970 32069 ts_tablet_manager.cc:1367] T 37456bdef9f94a52b63d0ab75f8ffa8b P 64688d2615e44408ab868cb531540a84: Time spent bootstrapping tablet: real 0.012s	user 0.005s	sys 0.005s
I20240502 02:01:55.874878 32069 raft_consensus.cc:348] T 37456bdef9f94a52b63d0ab75f8ffa8b P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:55.875136 32069 raft_consensus.cc:374] T 37456bdef9f94a52b63d0ab75f8ffa8b P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:01:55.875219 32069 raft_consensus.cc:729] T 37456bdef9f94a52b63d0ab75f8ffa8b P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64688d2615e44408ab868cb531540a84, State: Initialized, Role: FOLLOWER
I20240502 02:01:55.875473 32069 consensus_queue.cc:260] T 37456bdef9f94a52b63d0ab75f8ffa8b P 64688d2615e44408ab868cb531540a84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:55.876132 32069 ts_tablet_manager.cc:1398] T 37456bdef9f94a52b63d0ab75f8ffa8b P 64688d2615e44408ab868cb531540a84: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 02:01:55.881500 32072 tablet_bootstrap.cc:492] T 1b97531b47d44bf8a7f905ef339f857a P e325c0c7447f4b329994aafdee580666: No bootstrap required, opened a new log
I20240502 02:01:55.881498 32070 tablet_bootstrap.cc:492] T 9676b44f15e84ae28a64caa71f5722c6 P 0db0855dad46416094c68fc878712cbb: No bootstrap required, opened a new log
I20240502 02:01:55.881721 32072 ts_tablet_manager.cc:1367] T 1b97531b47d44bf8a7f905ef339f857a P e325c0c7447f4b329994aafdee580666: Time spent bootstrapping tablet: real 0.020s	user 0.011s	sys 0.008s
I20240502 02:01:55.881752 32070 ts_tablet_manager.cc:1367] T 9676b44f15e84ae28a64caa71f5722c6 P 0db0855dad46416094c68fc878712cbb: Time spent bootstrapping tablet: real 0.019s	user 0.003s	sys 0.012s
I20240502 02:01:55.882560 32070 raft_consensus.cc:348] T 9676b44f15e84ae28a64caa71f5722c6 P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:55.882563 32072 raft_consensus.cc:348] T 1b97531b47d44bf8a7f905ef339f857a P e325c0c7447f4b329994aafdee580666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:55.882818 32070 raft_consensus.cc:374] T 9676b44f15e84ae28a64caa71f5722c6 P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:01:55.882818 32072 raft_consensus.cc:374] T 1b97531b47d44bf8a7f905ef339f857a P e325c0c7447f4b329994aafdee580666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:01:55.882900 32070 raft_consensus.cc:729] T 9676b44f15e84ae28a64caa71f5722c6 P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0db0855dad46416094c68fc878712cbb, State: Initialized, Role: FOLLOWER
I20240502 02:01:55.882900 32072 raft_consensus.cc:729] T 1b97531b47d44bf8a7f905ef339f857a P e325c0c7447f4b329994aafdee580666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e325c0c7447f4b329994aafdee580666, State: Initialized, Role: FOLLOWER
I20240502 02:01:55.883168 32072 consensus_queue.cc:260] T 1b97531b47d44bf8a7f905ef339f857a P e325c0c7447f4b329994aafdee580666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:55.883167 32070 consensus_queue.cc:260] T 9676b44f15e84ae28a64caa71f5722c6 P 0db0855dad46416094c68fc878712cbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:55.883735 32070 ts_tablet_manager.cc:1398] T 9676b44f15e84ae28a64caa71f5722c6 P 0db0855dad46416094c68fc878712cbb: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 02:01:55.883742 32072 ts_tablet_manager.cc:1398] T 1b97531b47d44bf8a7f905ef339f857a P e325c0c7447f4b329994aafdee580666: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 02:01:55.884096 32072 tablet_bootstrap.cc:492] T 37456bdef9f94a52b63d0ab75f8ffa8b P e325c0c7447f4b329994aafdee580666: Bootstrap starting.
I20240502 02:01:55.884099 32070 tablet_bootstrap.cc:492] T 37456bdef9f94a52b63d0ab75f8ffa8b P 0db0855dad46416094c68fc878712cbb: Bootstrap starting.
I20240502 02:01:55.885794 32072 tablet_bootstrap.cc:654] T 37456bdef9f94a52b63d0ab75f8ffa8b P e325c0c7447f4b329994aafdee580666: Neither blocks nor log segments found. Creating new log.
I20240502 02:01:55.885790 32070 tablet_bootstrap.cc:654] T 37456bdef9f94a52b63d0ab75f8ffa8b P 0db0855dad46416094c68fc878712cbb: Neither blocks nor log segments found. Creating new log.
I20240502 02:01:55.887841 32072 tablet_bootstrap.cc:492] T 37456bdef9f94a52b63d0ab75f8ffa8b P e325c0c7447f4b329994aafdee580666: No bootstrap required, opened a new log
I20240502 02:01:55.888059 32070 tablet_bootstrap.cc:492] T 37456bdef9f94a52b63d0ab75f8ffa8b P 0db0855dad46416094c68fc878712cbb: No bootstrap required, opened a new log
I20240502 02:01:55.888056 32072 ts_tablet_manager.cc:1367] T 37456bdef9f94a52b63d0ab75f8ffa8b P e325c0c7447f4b329994aafdee580666: Time spent bootstrapping tablet: real 0.004s	user 0.002s	sys 0.002s
I20240502 02:01:55.888211 32070 ts_tablet_manager.cc:1367] T 37456bdef9f94a52b63d0ab75f8ffa8b P 0db0855dad46416094c68fc878712cbb: Time spent bootstrapping tablet: real 0.004s	user 0.002s	sys 0.002s
I20240502 02:01:55.888901 32072 raft_consensus.cc:348] T 37456bdef9f94a52b63d0ab75f8ffa8b P e325c0c7447f4b329994aafdee580666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:55.889132 32072 raft_consensus.cc:374] T 37456bdef9f94a52b63d0ab75f8ffa8b P e325c0c7447f4b329994aafdee580666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:01:55.889221 32072 raft_consensus.cc:729] T 37456bdef9f94a52b63d0ab75f8ffa8b P e325c0c7447f4b329994aafdee580666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e325c0c7447f4b329994aafdee580666, State: Initialized, Role: FOLLOWER
I20240502 02:01:55.888979 32070 raft_consensus.cc:348] T 37456bdef9f94a52b63d0ab75f8ffa8b P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:55.889395 32070 raft_consensus.cc:374] T 37456bdef9f94a52b63d0ab75f8ffa8b P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:01:55.889505 32070 raft_consensus.cc:729] T 37456bdef9f94a52b63d0ab75f8ffa8b P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0db0855dad46416094c68fc878712cbb, State: Initialized, Role: FOLLOWER
I20240502 02:01:55.889463 32072 consensus_queue.cc:260] T 37456bdef9f94a52b63d0ab75f8ffa8b P e325c0c7447f4b329994aafdee580666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:55.889968 32072 ts_tablet_manager.cc:1398] T 37456bdef9f94a52b63d0ab75f8ffa8b P e325c0c7447f4b329994aafdee580666: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 02:01:55.889782 32070 consensus_queue.cc:260] T 37456bdef9f94a52b63d0ab75f8ffa8b P 0db0855dad46416094c68fc878712cbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:55.890664 32070 ts_tablet_manager.cc:1398] T 37456bdef9f94a52b63d0ab75f8ffa8b P 0db0855dad46416094c68fc878712cbb: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 02:01:55.905754 32061 raft_consensus.cc:482] T 1b97531b47d44bf8a7f905ef339f857a P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 02:01:55.905963 32061 raft_consensus.cc:504] T 1b97531b47d44bf8a7f905ef339f857a P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:55.906728 32061 leader_election.cc:288] T 1b97531b47d44bf8a7f905ef339f857a P 64688d2615e44408ab868cb531540a84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0db0855dad46416094c68fc878712cbb (127.25.109.130:38025), e325c0c7447f4b329994aafdee580666 (127.25.109.131:43937)
I20240502 02:01:55.907387 31558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b97531b47d44bf8a7f905ef339f857a" candidate_uuid: "64688d2615e44408ab868cb531540a84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e325c0c7447f4b329994aafdee580666" is_pre_election: true
I20240502 02:01:55.907387 31428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b97531b47d44bf8a7f905ef339f857a" candidate_uuid: "64688d2615e44408ab868cb531540a84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0db0855dad46416094c68fc878712cbb" is_pre_election: true
I20240502 02:01:55.907665 31558 raft_consensus.cc:2454] T 1b97531b47d44bf8a7f905ef339f857a P e325c0c7447f4b329994aafdee580666 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64688d2615e44408ab868cb531540a84 in term 0.
I20240502 02:01:55.907665 31428 raft_consensus.cc:2454] T 1b97531b47d44bf8a7f905ef339f857a P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64688d2615e44408ab868cb531540a84 in term 0.
I20240502 02:01:55.908165 31226 leader_election.cc:302] T 1b97531b47d44bf8a7f905ef339f857a P 64688d2615e44408ab868cb531540a84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0db0855dad46416094c68fc878712cbb, 64688d2615e44408ab868cb531540a84, e325c0c7447f4b329994aafdee580666; no voters: 
I20240502 02:01:55.908481 32061 raft_consensus.cc:2781] T 1b97531b47d44bf8a7f905ef339f857a P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 02:01:55.908607 32061 raft_consensus.cc:482] T 1b97531b47d44bf8a7f905ef339f857a P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 02:01:55.908692 32061 raft_consensus.cc:3037] T 1b97531b47d44bf8a7f905ef339f857a P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:01:55.910733 32061 raft_consensus.cc:504] T 1b97531b47d44bf8a7f905ef339f857a P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:55.911370 32061 leader_election.cc:288] T 1b97531b47d44bf8a7f905ef339f857a P 64688d2615e44408ab868cb531540a84 [CANDIDATE]: Term 1 election: Requested vote from peers 0db0855dad46416094c68fc878712cbb (127.25.109.130:38025), e325c0c7447f4b329994aafdee580666 (127.25.109.131:43937)
I20240502 02:01:55.911975 31558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b97531b47d44bf8a7f905ef339f857a" candidate_uuid: "64688d2615e44408ab868cb531540a84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e325c0c7447f4b329994aafdee580666"
I20240502 02:01:55.911959 31428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b97531b47d44bf8a7f905ef339f857a" candidate_uuid: "64688d2615e44408ab868cb531540a84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0db0855dad46416094c68fc878712cbb"
I20240502 02:01:55.912189 31558 raft_consensus.cc:3037] T 1b97531b47d44bf8a7f905ef339f857a P e325c0c7447f4b329994aafdee580666 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:01:55.912186 31428 raft_consensus.cc:3037] T 1b97531b47d44bf8a7f905ef339f857a P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:01:55.913089 32059 raft_consensus.cc:482] T 37456bdef9f94a52b63d0ab75f8ffa8b P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 02:01:55.913288 32059 raft_consensus.cc:504] T 37456bdef9f94a52b63d0ab75f8ffa8b P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:55.914173 32059 leader_election.cc:288] T 37456bdef9f94a52b63d0ab75f8ffa8b P 0db0855dad46416094c68fc878712cbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64688d2615e44408ab868cb531540a84 (127.25.109.129:33021), e325c0c7447f4b329994aafdee580666 (127.25.109.131:43937)
I20240502 02:01:55.914598 31428 raft_consensus.cc:2454] T 1b97531b47d44bf8a7f905ef339f857a P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64688d2615e44408ab868cb531540a84 in term 1.
I20240502 02:01:55.914788 31558 raft_consensus.cc:2454] T 1b97531b47d44bf8a7f905ef339f857a P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64688d2615e44408ab868cb531540a84 in term 1.
I20240502 02:01:55.914883 31559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37456bdef9f94a52b63d0ab75f8ffa8b" candidate_uuid: "0db0855dad46416094c68fc878712cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e325c0c7447f4b329994aafdee580666" is_pre_election: true
I20240502 02:01:55.914898 31292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37456bdef9f94a52b63d0ab75f8ffa8b" candidate_uuid: "0db0855dad46416094c68fc878712cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64688d2615e44408ab868cb531540a84" is_pre_election: true
I20240502 02:01:55.915140 31559 raft_consensus.cc:2454] T 37456bdef9f94a52b63d0ab75f8ffa8b P e325c0c7447f4b329994aafdee580666 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0db0855dad46416094c68fc878712cbb in term 0.
I20240502 02:01:55.915141 31292 raft_consensus.cc:2454] T 37456bdef9f94a52b63d0ab75f8ffa8b P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0db0855dad46416094c68fc878712cbb in term 0.
I20240502 02:01:55.915335 31226 leader_election.cc:302] T 1b97531b47d44bf8a7f905ef339f857a P 64688d2615e44408ab868cb531540a84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64688d2615e44408ab868cb531540a84, e325c0c7447f4b329994aafdee580666; no voters: 
I20240502 02:01:55.915534 31361 leader_election.cc:302] T 37456bdef9f94a52b63d0ab75f8ffa8b P 0db0855dad46416094c68fc878712cbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0db0855dad46416094c68fc878712cbb, e325c0c7447f4b329994aafdee580666; no voters: 
I20240502 02:01:55.915658 32061 raft_consensus.cc:2781] T 1b97531b47d44bf8a7f905ef339f857a P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:01:55.915796 32061 raft_consensus.cc:686] T 1b97531b47d44bf8a7f905ef339f857a P 64688d2615e44408ab868cb531540a84 [term 1 LEADER]: Becoming Leader. State: Replica: 64688d2615e44408ab868cb531540a84, State: Running, Role: LEADER
I20240502 02:01:55.915803 32059 raft_consensus.cc:2781] T 37456bdef9f94a52b63d0ab75f8ffa8b P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 02:01:55.915942 32059 raft_consensus.cc:482] T 37456bdef9f94a52b63d0ab75f8ffa8b P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 02:01:55.916054 32059 raft_consensus.cc:3037] T 37456bdef9f94a52b63d0ab75f8ffa8b P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:01:55.916322 32061 consensus_queue.cc:237] T 1b97531b47d44bf8a7f905ef339f857a P 64688d2615e44408ab868cb531540a84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:55.919391 32059 raft_consensus.cc:504] T 37456bdef9f94a52b63d0ab75f8ffa8b P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:55.920003 32059 leader_election.cc:288] T 37456bdef9f94a52b63d0ab75f8ffa8b P 0db0855dad46416094c68fc878712cbb [CANDIDATE]: Term 1 election: Requested vote from peers 64688d2615e44408ab868cb531540a84 (127.25.109.129:33021), e325c0c7447f4b329994aafdee580666 (127.25.109.131:43937)
I20240502 02:01:55.920603 31559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37456bdef9f94a52b63d0ab75f8ffa8b" candidate_uuid: "0db0855dad46416094c68fc878712cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e325c0c7447f4b329994aafdee580666"
I20240502 02:01:55.920612 31292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37456bdef9f94a52b63d0ab75f8ffa8b" candidate_uuid: "0db0855dad46416094c68fc878712cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64688d2615e44408ab868cb531540a84"
I20240502 02:01:55.920805 31292 raft_consensus.cc:3037] T 37456bdef9f94a52b63d0ab75f8ffa8b P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:01:55.920826 31559 raft_consensus.cc:3037] T 37456bdef9f94a52b63d0ab75f8ffa8b P e325c0c7447f4b329994aafdee580666 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:01:55.920750 31147 catalog_manager.cc:5484] T 1b97531b47d44bf8a7f905ef339f857a P 64688d2615e44408ab868cb531540a84 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64688d2615e44408ab868cb531540a84 (127.25.109.129). New cstate: current_term: 1 leader_uuid: "64688d2615e44408ab868cb531540a84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20240502 02:01:55.923595 31559 raft_consensus.cc:2454] T 37456bdef9f94a52b63d0ab75f8ffa8b P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0db0855dad46416094c68fc878712cbb in term 1.
I20240502 02:01:55.923595 31292 raft_consensus.cc:2454] T 37456bdef9f94a52b63d0ab75f8ffa8b P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0db0855dad46416094c68fc878712cbb in term 1.
I20240502 02:01:55.924058 31361 leader_election.cc:302] T 37456bdef9f94a52b63d0ab75f8ffa8b P 0db0855dad46416094c68fc878712cbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0db0855dad46416094c68fc878712cbb, e325c0c7447f4b329994aafdee580666; no voters: 
I20240502 02:01:55.924400 32059 raft_consensus.cc:2781] T 37456bdef9f94a52b63d0ab75f8ffa8b P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:01:55.924527 32059 raft_consensus.cc:686] T 37456bdef9f94a52b63d0ab75f8ffa8b P 0db0855dad46416094c68fc878712cbb [term 1 LEADER]: Becoming Leader. State: Replica: 0db0855dad46416094c68fc878712cbb, State: Running, Role: LEADER
I20240502 02:01:55.924785 32059 consensus_queue.cc:237] T 37456bdef9f94a52b63d0ab75f8ffa8b P 0db0855dad46416094c68fc878712cbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:55.927623 31147 catalog_manager.cc:5484] T 37456bdef9f94a52b63d0ab75f8ffa8b P 0db0855dad46416094c68fc878712cbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0db0855dad46416094c68fc878712cbb (127.25.109.130). New cstate: current_term: 1 leader_uuid: "0db0855dad46416094c68fc878712cbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20240502 02:01:56.319296 32061 consensus_queue.cc:1035] T 1b97531b47d44bf8a7f905ef339f857a P 64688d2615e44408ab868cb531540a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:01:56.319991 32061 raft_consensus.cc:482] T 9676b44f15e84ae28a64caa71f5722c6 P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 02:01:56.320147 32061 raft_consensus.cc:504] T 9676b44f15e84ae28a64caa71f5722c6 P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:56.320888 32061 leader_election.cc:288] T 9676b44f15e84ae28a64caa71f5722c6 P 64688d2615e44408ab868cb531540a84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0db0855dad46416094c68fc878712cbb (127.25.109.130:38025), e325c0c7447f4b329994aafdee580666 (127.25.109.131:43937)
I20240502 02:01:56.321239 31427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9676b44f15e84ae28a64caa71f5722c6" candidate_uuid: "64688d2615e44408ab868cb531540a84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0db0855dad46416094c68fc878712cbb" is_pre_election: true
I20240502 02:01:56.321509 31427 raft_consensus.cc:2454] T 9676b44f15e84ae28a64caa71f5722c6 P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64688d2615e44408ab868cb531540a84 in term 0.
I20240502 02:01:56.322126 31227 leader_election.cc:302] T 9676b44f15e84ae28a64caa71f5722c6 P 64688d2615e44408ab868cb531540a84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0db0855dad46416094c68fc878712cbb, 64688d2615e44408ab868cb531540a84; no voters: 
I20240502 02:01:56.322497 32061 raft_consensus.cc:2781] T 9676b44f15e84ae28a64caa71f5722c6 P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 02:01:56.322639 32061 raft_consensus.cc:482] T 9676b44f15e84ae28a64caa71f5722c6 P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 02:01:56.322728 32061 raft_consensus.cc:3037] T 9676b44f15e84ae28a64caa71f5722c6 P 64688d2615e44408ab868cb531540a84 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:01:56.323241 32059 consensus_queue.cc:1035] T 37456bdef9f94a52b63d0ab75f8ffa8b P 0db0855dad46416094c68fc878712cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:01:56.323280 31559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9676b44f15e84ae28a64caa71f5722c6" candidate_uuid: "64688d2615e44408ab868cb531540a84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e325c0c7447f4b329994aafdee580666" is_pre_election: true
I20240502 02:01:56.323501 31559 raft_consensus.cc:2454] T 9676b44f15e84ae28a64caa71f5722c6 P e325c0c7447f4b329994aafdee580666 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64688d2615e44408ab868cb531540a84 in term 0.
I20240502 02:01:56.325394 32061 raft_consensus.cc:504] T 9676b44f15e84ae28a64caa71f5722c6 P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:56.326766 31427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9676b44f15e84ae28a64caa71f5722c6" candidate_uuid: "64688d2615e44408ab868cb531540a84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0db0855dad46416094c68fc878712cbb"
I20240502 02:01:56.326812 31558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9676b44f15e84ae28a64caa71f5722c6" candidate_uuid: "64688d2615e44408ab868cb531540a84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e325c0c7447f4b329994aafdee580666"
I20240502 02:01:56.326992 31427 raft_consensus.cc:3037] T 9676b44f15e84ae28a64caa71f5722c6 P 0db0855dad46416094c68fc878712cbb [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:01:56.327016 31558 raft_consensus.cc:3037] T 9676b44f15e84ae28a64caa71f5722c6 P e325c0c7447f4b329994aafdee580666 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:01:56.329847 31427 raft_consensus.cc:2454] T 9676b44f15e84ae28a64caa71f5722c6 P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64688d2615e44408ab868cb531540a84 in term 1.
I20240502 02:01:56.329921 31558 raft_consensus.cc:2454] T 9676b44f15e84ae28a64caa71f5722c6 P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64688d2615e44408ab868cb531540a84 in term 1.
I20240502 02:01:56.330432 31227 leader_election.cc:302] T 9676b44f15e84ae28a64caa71f5722c6 P 64688d2615e44408ab868cb531540a84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0db0855dad46416094c68fc878712cbb, 64688d2615e44408ab868cb531540a84; no voters: 
I20240502 02:01:56.330498 32061 leader_election.cc:288] T 9676b44f15e84ae28a64caa71f5722c6 P 64688d2615e44408ab868cb531540a84 [CANDIDATE]: Term 1 election: Requested vote from peers 0db0855dad46416094c68fc878712cbb (127.25.109.130:38025), e325c0c7447f4b329994aafdee580666 (127.25.109.131:43937)
I20240502 02:01:56.330955 32061 raft_consensus.cc:2781] T 9676b44f15e84ae28a64caa71f5722c6 P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:01:56.331125 32061 raft_consensus.cc:686] T 9676b44f15e84ae28a64caa71f5722c6 P 64688d2615e44408ab868cb531540a84 [term 1 LEADER]: Becoming Leader. State: Replica: 64688d2615e44408ab868cb531540a84, State: Running, Role: LEADER
I20240502 02:01:56.331492 32061 consensus_queue.cc:237] T 9676b44f15e84ae28a64caa71f5722c6 P 64688d2615e44408ab868cb531540a84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } }
I20240502 02:01:56.335968 32084 consensus_queue.cc:1035] T 1b97531b47d44bf8a7f905ef339f857a P 64688d2615e44408ab868cb531540a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:01:56.345247 32059 consensus_queue.cc:1035] T 37456bdef9f94a52b63d0ab75f8ffa8b P 0db0855dad46416094c68fc878712cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:01:56.346907 31141 catalog_manager.cc:5484] T 9676b44f15e84ae28a64caa71f5722c6 P 64688d2615e44408ab868cb531540a84 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64688d2615e44408ab868cb531540a84 (127.25.109.129). New cstate: current_term: 1 leader_uuid: "64688d2615e44408ab868cb531540a84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20240502 02:01:56.752882 32061 consensus_queue.cc:1035] T 9676b44f15e84ae28a64caa71f5722c6 P 64688d2615e44408ab868cb531540a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:01:56.757622 31928 consensus_queue.cc:1035] T 9676b44f15e84ae28a64caa71f5722c6 P 64688d2615e44408ab868cb531540a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 02:01:56.764482 32094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:01:56.764884 32094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:01:56.775055 32094 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
W20240502 02:01:56.967808 31361 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.109.131:43937, user_credentials={real_user=slave}} blocked reactor thread for 59403.1us

INSERT report
    rows total: 2000
    time total: 180.146 ms
  time per row: 0.0900728 ms
I20240502 02:01:57.120254 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.132:38023
--local_ip_for_outbound_sockets=127.25.109.132
--tserver_master_addrs=127.25.109.190:43455
--webserver_port=35619
--webserver_interface=127.25.109.132
--builtin_ntp_servers=127.25.109.148:42757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 02:01:57.478073 32146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:01:57.478391 32146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 02:01:57.478528 32146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:01:57.486593 32146 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 02:01:57.486752 32146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 02:01:57.486907 32146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.132
I20240502 02:01:57.496524 32146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:42757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.132:38023
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.25.109.132
--webserver_port=35619
--tserver_master_addrs=127.25.109.190:43455
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.132
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 02:01:57.497951 32146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 02:01:57.499812 32146 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 02:01:57.512910 32156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:01:57.512910 32151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 02:01:57.513353 32146 server_base.cc:1034] running on GCE node
W20240502 02:01:57.513710 32152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 02:01:57.514462 32146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 02:01:57.515472 32146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 02:01:57.516672 32146 hybrid_clock.cc:648] HybridClock initialized: now 1714615317516621 us; error 43 us; skew 500 ppm
I20240502 02:01:57.517014 32146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 02:01:57.519294 32146 webserver.cc:460] Webserver started at http://127.25.109.132:35619/ using document root <none> and password file <none>
I20240502 02:01:57.520085 32146 fs_manager.cc:359] Metadata directory not provided
I20240502 02:01:57.520220 32146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 02:01:57.525554 32146 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.004s	sys 0.000s
I20240502 02:01:57.528290 32162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:01:57.529575 32146 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 02:01:57.529736 32146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "29755ed6e9444b2daf5807bd23451d29"
format_stamp: "Formatted at 2024-05-02 02:01:48 on dist-test-slave-kj4b"
I20240502 02:01:57.530606 32146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 02:01:57.560003 32146 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 02:01:57.560897 32146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 02:01:57.561188 32146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 02:01:57.562265 32146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 02:01:57.564025 32146 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 02:01:57.564118 32146 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:01:57.564247 32146 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 02:01:57.564301 32146 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:01:57.595511 32146 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.132:38023
I20240502 02:01:57.595561 32274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.132:38023 every 8 connection(s)
I20240502 02:01:57.597136 32146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 02:01:57.606909 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 32146
I20240502 02:01:57.607656 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.133:36143
--local_ip_for_outbound_sockets=127.25.109.133
--tserver_master_addrs=127.25.109.190:43455
--webserver_port=37013
--webserver_interface=127.25.109.133
--builtin_ntp_servers=127.25.109.148:42757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 02:01:57.616245 32275 heartbeater.cc:347] Connected to a master server at 127.25.109.190:43455
I20240502 02:01:57.616653 32275 heartbeater.cc:464] Registering TS with master...
I20240502 02:01:57.617717 32275 heartbeater.cc:511] Master 127.25.109.190:43455 requested a full tablet report, sending...
I20240502 02:01:57.619784 31141 ts_manager.cc:194] Re-registered known tserver with Master: 29755ed6e9444b2daf5807bd23451d29 (127.25.109.132:38023)
I20240502 02:01:57.620968 31141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.132:48395
W20240502 02:01:57.974671 32279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:01:57.974978 32279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 02:01:57.975116 32279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:01:57.983403 32279 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 02:01:57.983543 32279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 02:01:57.983701 32279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.133
I20240502 02:01:57.993297 32279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:42757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.133:36143
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.25.109.133
--webserver_port=37013
--tserver_master_addrs=127.25.109.190:43455
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.133
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 02:01:57.994709 32279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 02:01:57.996973 32279 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 02:01:58.009850 32284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:01:58.009876 32287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:01:58.010242 32285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 02:01:58.010713 32279 server_base.cc:1034] running on GCE node
I20240502 02:01:58.011333 32279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 02:01:58.012398 32279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 02:01:58.013618 32279 hybrid_clock.cc:648] HybridClock initialized: now 1714615318013558 us; error 54 us; skew 500 ppm
I20240502 02:01:58.013988 32279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 02:01:58.016849 32279 webserver.cc:460] Webserver started at http://127.25.109.133:37013/ using document root <none> and password file <none>
I20240502 02:01:58.017696 32279 fs_manager.cc:359] Metadata directory not provided
I20240502 02:01:58.017843 32279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 02:01:58.023932 32279 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240502 02:01:58.030196 32293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:01:58.031821 32279 fs_manager.cc:725] Time spent opening block manager: real 0.006s	user 0.004s	sys 0.001s
I20240502 02:01:58.032060 32279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "7da2c0838fce40c5b5000c4d71c72708"
format_stamp: "Formatted at 2024-05-02 02:01:49 on dist-test-slave-kj4b"
I20240502 02:01:58.033035 32279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 02:01:58.049247 32279 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 02:01:58.050185 32279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 02:01:58.050594 32279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 02:01:58.051630 32279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 02:01:58.053386 32279 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 02:01:58.053472 32279 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:01:58.053604 32279 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 02:01:58.053658 32279 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:01:58.084630 32279 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.133:36143
I20240502 02:01:58.084671 32408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.133:36143 every 8 connection(s)
I20240502 02:01:58.086210 32279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 02:01:58.093683 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 32279
I20240502 02:01:58.101003 32409 heartbeater.cc:347] Connected to a master server at 127.25.109.190:43455
I20240502 02:01:58.101395 32409 heartbeater.cc:464] Registering TS with master...
I20240502 02:01:58.102473 32409 heartbeater.cc:511] Master 127.25.109.190:43455 requested a full tablet report, sending...
I20240502 02:01:58.104226 31141 ts_manager.cc:194] Re-registered known tserver with Master: 7da2c0838fce40c5b5000c4d71c72708 (127.25.109.133:36143)
I20240502 02:01:58.105084 31141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.133:59263
I20240502 02:01:58.607251 32210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:58.624118 32275 heartbeater.cc:503] Master 127.25.109.190:43455 was elected leader, sending a full tablet report...
I20240502 02:01:58.626214 31407 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:01:58.636477 31270 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:01:58.649340 32343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:58.663847 31538 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:58.828249 31407 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:01:58.836196 32210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:58.844214 31270 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:01:58.847684 32343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:58.859964 31538 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:59.037889 31407 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:01:59.038743 32210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:59.050210 31270 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:01:59.066900 32343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:59.086756 31538 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:59.107982 32409 heartbeater.cc:503] Master 127.25.109.190:43455 was elected leader, sending a full tablet report...
I20240502 02:01:59.234862 32210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:59.237916 31407 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:01:59.249264 32343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:01:59.267113 31270 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:01:59.280129 31538 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:01:59.340811 31291 consensus_queue.cc:237] T 1b97531b47d44bf8a7f905ef339f857a P 64688d2615e44408ab868cb531540a84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: true } }
I20240502 02:01:59.343494 31428 raft_consensus.cc:1261] T 1b97531b47d44bf8a7f905ef339f857a P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Refusing update from remote peer 64688d2615e44408ab868cb531540a84: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 02:01:59.343637 31558 raft_consensus.cc:1261] T 1b97531b47d44bf8a7f905ef339f857a P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Refusing update from remote peer 64688d2615e44408ab868cb531540a84: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 02:01:59.344359 32134 consensus_queue.cc:1035] T 1b97531b47d44bf8a7f905ef339f857a P 64688d2615e44408ab868cb531540a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:01:59.345045 32522 consensus_queue.cc:1035] T 1b97531b47d44bf8a7f905ef339f857a P 64688d2615e44408ab868cb531540a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:01:59.348676 32134 raft_consensus.cc:2932] T 1b97531b47d44bf8a7f905ef339f857a P 64688d2615e44408ab868cb531540a84 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 29755ed6e9444b2daf5807bd23451d29 (127.25.109.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: true } } }
I20240502 02:01:59.349462 31428 raft_consensus.cc:2932] T 1b97531b47d44bf8a7f905ef339f857a P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 29755ed6e9444b2daf5807bd23451d29 (127.25.109.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: true } } }
I20240502 02:01:59.349599 31558 raft_consensus.cc:2932] T 1b97531b47d44bf8a7f905ef339f857a P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 29755ed6e9444b2daf5807bd23451d29 (127.25.109.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: true } } }
W20240502 02:01:59.353315 31225 consensus_peers.cc:485] T 1b97531b47d44bf8a7f905ef339f857a P 64688d2615e44408ab868cb531540a84 -> Peer 29755ed6e9444b2daf5807bd23451d29 (127.25.109.132:38023): Couldn't send request to peer 29755ed6e9444b2daf5807bd23451d29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1b97531b47d44bf8a7f905ef339f857a. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:01:59.355561 31144 catalog_manager.cc:5484] T 1b97531b47d44bf8a7f905ef339f857a P e325c0c7447f4b329994aafdee580666 reported cstate change: config changed from index -1 to 4, NON_VOTER 29755ed6e9444b2daf5807bd23451d29 (127.25.109.132) added. New cstate: current_term: 1 leader_uuid: "64688d2615e44408ab868cb531540a84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: true } } }
I20240502 02:01:59.369805 31428 consensus_queue.cc:237] T bec2e393d5e841c7857d6fd092aac53e P 0db0855dad46416094c68fc878712cbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: true } }
I20240502 02:01:59.372455 31291 raft_consensus.cc:1261] T bec2e393d5e841c7857d6fd092aac53e P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Refusing update from remote peer 0db0855dad46416094c68fc878712cbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 02:01:59.373550 32572 consensus_queue.cc:1035] T bec2e393d5e841c7857d6fd092aac53e P 0db0855dad46416094c68fc878712cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:01:59.373901 31558 raft_consensus.cc:1261] T bec2e393d5e841c7857d6fd092aac53e P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Refusing update from remote peer 0db0855dad46416094c68fc878712cbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 02:01:59.374466 32572 consensus_queue.cc:1035] T bec2e393d5e841c7857d6fd092aac53e P 0db0855dad46416094c68fc878712cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:01:59.379448 32128 raft_consensus.cc:2932] T bec2e393d5e841c7857d6fd092aac53e P 0db0855dad46416094c68fc878712cbb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7da2c0838fce40c5b5000c4d71c72708 (127.25.109.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: true } } }
I20240502 02:01:59.380110 31291 raft_consensus.cc:2932] T bec2e393d5e841c7857d6fd092aac53e P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7da2c0838fce40c5b5000c4d71c72708 (127.25.109.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: true } } }
I20240502 02:01:59.380375 31558 raft_consensus.cc:2932] T bec2e393d5e841c7857d6fd092aac53e P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7da2c0838fce40c5b5000c4d71c72708 (127.25.109.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: true } } }
W20240502 02:01:59.383899 31361 consensus_peers.cc:485] T bec2e393d5e841c7857d6fd092aac53e P 0db0855dad46416094c68fc878712cbb -> Peer 7da2c0838fce40c5b5000c4d71c72708 (127.25.109.133:36143): Couldn't send request to peer 7da2c0838fce40c5b5000c4d71c72708. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bec2e393d5e841c7857d6fd092aac53e. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:01:59.385677 31141 catalog_manager.cc:5484] T bec2e393d5e841c7857d6fd092aac53e P 0db0855dad46416094c68fc878712cbb reported cstate change: config changed from index -1 to 4, NON_VOTER 7da2c0838fce40c5b5000c4d71c72708 (127.25.109.133) added. New cstate: current_term: 1 leader_uuid: "0db0855dad46416094c68fc878712cbb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 02:01:59.398300 31291 consensus_queue.cc:237] T a10fd1feee3a4838a8fba1599a1b890d P 64688d2615e44408ab868cb531540a84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: true } }
I20240502 02:01:59.400960 31558 raft_consensus.cc:1261] T a10fd1feee3a4838a8fba1599a1b890d P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Refusing update from remote peer 64688d2615e44408ab868cb531540a84: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 02:01:59.401078 31225 consensus_peers.cc:485] T a10fd1feee3a4838a8fba1599a1b890d P 64688d2615e44408ab868cb531540a84 -> Peer 29755ed6e9444b2daf5807bd23451d29 (127.25.109.132:38023): Couldn't send request to peer 29755ed6e9444b2daf5807bd23451d29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a10fd1feee3a4838a8fba1599a1b890d. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:01:59.401531 32522 consensus_queue.cc:1035] T a10fd1feee3a4838a8fba1599a1b890d P 64688d2615e44408ab868cb531540a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:01:59.401616 31428 raft_consensus.cc:1261] T a10fd1feee3a4838a8fba1599a1b890d P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Refusing update from remote peer 64688d2615e44408ab868cb531540a84: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 02:01:59.402159 32557 consensus_queue.cc:1035] T a10fd1feee3a4838a8fba1599a1b890d P 64688d2615e44408ab868cb531540a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:01:59.404304 32522 raft_consensus.cc:2932] T a10fd1feee3a4838a8fba1599a1b890d P 64688d2615e44408ab868cb531540a84 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 29755ed6e9444b2daf5807bd23451d29 (127.25.109.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: true } } }
I20240502 02:01:59.405668 31428 raft_consensus.cc:2932] T a10fd1feee3a4838a8fba1599a1b890d P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 29755ed6e9444b2daf5807bd23451d29 (127.25.109.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: true } } }
I20240502 02:01:59.408407 31558 raft_consensus.cc:2932] T a10fd1feee3a4838a8fba1599a1b890d P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 29755ed6e9444b2daf5807bd23451d29 (127.25.109.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: true } } }
I20240502 02:01:59.411274 31150 catalog_manager.cc:5484] T a10fd1feee3a4838a8fba1599a1b890d P 0db0855dad46416094c68fc878712cbb reported cstate change: config changed from index -1 to 4, NON_VOTER 29755ed6e9444b2daf5807bd23451d29 (127.25.109.132) added. New cstate: current_term: 1 leader_uuid: "64688d2615e44408ab868cb531540a84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: true } } }
I20240502 02:01:59.429459 31291 consensus_queue.cc:237] T 171f811ca5584cd3ab08fc25a9b7f7fc P 64688d2615e44408ab868cb531540a84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: true } }
I20240502 02:01:59.432179 31558 raft_consensus.cc:1261] T 171f811ca5584cd3ab08fc25a9b7f7fc P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Refusing update from remote peer 64688d2615e44408ab868cb531540a84: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 02:01:59.432170 31428 raft_consensus.cc:1261] T 171f811ca5584cd3ab08fc25a9b7f7fc P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Refusing update from remote peer 64688d2615e44408ab868cb531540a84: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 02:01:59.432835 32557 consensus_queue.cc:1035] T 171f811ca5584cd3ab08fc25a9b7f7fc P 64688d2615e44408ab868cb531540a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:01:59.433197 32134 consensus_queue.cc:1035] T 171f811ca5584cd3ab08fc25a9b7f7fc P 64688d2615e44408ab868cb531540a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:01:59.437655 32134 raft_consensus.cc:2932] T 171f811ca5584cd3ab08fc25a9b7f7fc P 64688d2615e44408ab868cb531540a84 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7da2c0838fce40c5b5000c4d71c72708 (127.25.109.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: true } } }
W20240502 02:01:59.439237 31226 consensus_peers.cc:485] T 171f811ca5584cd3ab08fc25a9b7f7fc P 64688d2615e44408ab868cb531540a84 -> Peer 7da2c0838fce40c5b5000c4d71c72708 (127.25.109.133:36143): Couldn't send request to peer 7da2c0838fce40c5b5000c4d71c72708. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 171f811ca5584cd3ab08fc25a9b7f7fc. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:01:59.438869 31558 raft_consensus.cc:2932] T 171f811ca5584cd3ab08fc25a9b7f7fc P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7da2c0838fce40c5b5000c4d71c72708 (127.25.109.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: true } } }
I20240502 02:01:59.442977 31428 raft_consensus.cc:2932] T 171f811ca5584cd3ab08fc25a9b7f7fc P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7da2c0838fce40c5b5000c4d71c72708 (127.25.109.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: true } } }
I20240502 02:01:59.445320 31141 catalog_manager.cc:5484] T 171f811ca5584cd3ab08fc25a9b7f7fc P e325c0c7447f4b329994aafdee580666 reported cstate change: config changed from index -1 to 4, NON_VOTER 7da2c0838fce40c5b5000c4d71c72708 (127.25.109.133) added. New cstate: current_term: 1 leader_uuid: "64688d2615e44408ab868cb531540a84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: true } } }
I20240502 02:01:59.459210 31428 consensus_queue.cc:237] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: true } }
I20240502 02:01:59.461858 31291 raft_consensus.cc:1261] T 28de9be2be73409387480db617b0031c P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Refusing update from remote peer 0db0855dad46416094c68fc878712cbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 02:01:59.462046 31558 raft_consensus.cc:1261] T 28de9be2be73409387480db617b0031c P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Refusing update from remote peer 0db0855dad46416094c68fc878712cbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 02:01:59.462558 32576 consensus_queue.cc:1035] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:01:59.462949 32128 consensus_queue.cc:1035] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:01:59.466414 32576 raft_consensus.cc:2932] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 29755ed6e9444b2daf5807bd23451d29 (127.25.109.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: true } } }
I20240502 02:01:59.467058 31558 raft_consensus.cc:2932] T 28de9be2be73409387480db617b0031c P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 29755ed6e9444b2daf5807bd23451d29 (127.25.109.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: true } } }
W20240502 02:01:59.468905 31360 consensus_peers.cc:485] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb -> Peer 29755ed6e9444b2daf5807bd23451d29 (127.25.109.132:38023): Couldn't send request to peer 29755ed6e9444b2daf5807bd23451d29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 28de9be2be73409387480db617b0031c. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:01:59.472244 31291 raft_consensus.cc:2932] T 28de9be2be73409387480db617b0031c P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 29755ed6e9444b2daf5807bd23451d29 (127.25.109.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: true } } }
I20240502 02:01:59.472754 31150 catalog_manager.cc:5484] T 28de9be2be73409387480db617b0031c P e325c0c7447f4b329994aafdee580666 reported cstate change: config changed from index -1 to 4, NON_VOTER 29755ed6e9444b2daf5807bd23451d29 (127.25.109.132) added. New cstate: current_term: 1 leader_uuid: "0db0855dad46416094c68fc878712cbb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: true } } }
I20240502 02:01:59.488229 31291 consensus_queue.cc:237] T 9676b44f15e84ae28a64caa71f5722c6 P 64688d2615e44408ab868cb531540a84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: true } }
I20240502 02:01:59.490818 31428 raft_consensus.cc:1261] T 9676b44f15e84ae28a64caa71f5722c6 P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Refusing update from remote peer 64688d2615e44408ab868cb531540a84: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 02:01:59.490803 31558 raft_consensus.cc:1261] T 9676b44f15e84ae28a64caa71f5722c6 P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Refusing update from remote peer 64688d2615e44408ab868cb531540a84: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 02:01:59.490995 31226 consensus_peers.cc:485] T 9676b44f15e84ae28a64caa71f5722c6 P 64688d2615e44408ab868cb531540a84 -> Peer 7da2c0838fce40c5b5000c4d71c72708 (127.25.109.133:36143): Couldn't send request to peer 7da2c0838fce40c5b5000c4d71c72708. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9676b44f15e84ae28a64caa71f5722c6. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:01:59.491415 32557 consensus_queue.cc:1035] T 9676b44f15e84ae28a64caa71f5722c6 P 64688d2615e44408ab868cb531540a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:01:59.491752 32134 consensus_queue.cc:1035] T 9676b44f15e84ae28a64caa71f5722c6 P 64688d2615e44408ab868cb531540a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:01:59.494832 32522 raft_consensus.cc:2932] T 9676b44f15e84ae28a64caa71f5722c6 P 64688d2615e44408ab868cb531540a84 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7da2c0838fce40c5b5000c4d71c72708 (127.25.109.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: true } } }
I20240502 02:01:59.495509 31558 raft_consensus.cc:2932] T 9676b44f15e84ae28a64caa71f5722c6 P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7da2c0838fce40c5b5000c4d71c72708 (127.25.109.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: true } } }
I20240502 02:01:59.495826 31428 raft_consensus.cc:2932] T 9676b44f15e84ae28a64caa71f5722c6 P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7da2c0838fce40c5b5000c4d71c72708 (127.25.109.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: true } } }
I20240502 02:01:59.500300 31150 catalog_manager.cc:5484] T 9676b44f15e84ae28a64caa71f5722c6 P 64688d2615e44408ab868cb531540a84 reported cstate change: config changed from index -1 to 4, NON_VOTER 7da2c0838fce40c5b5000c4d71c72708 (127.25.109.133) added. New cstate: current_term: 1 leader_uuid: "64688d2615e44408ab868cb531540a84" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 02:01:59.513849 31558 consensus_queue.cc:237] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: true } }
I20240502 02:01:59.516783 31428 raft_consensus.cc:1261] T 3475ca93c949435699b590bd97e3735c P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Refusing update from remote peer e325c0c7447f4b329994aafdee580666: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 02:01:59.517572 32616 consensus_queue.cc:1035] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:01:59.517855 31291 raft_consensus.cc:1261] T 3475ca93c949435699b590bd97e3735c P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Refusing update from remote peer e325c0c7447f4b329994aafdee580666: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 02:01:59.518786 32616 consensus_queue.cc:1035] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:01:59.521276 32617 raft_consensus.cc:2932] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7da2c0838fce40c5b5000c4d71c72708 (127.25.109.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: true } } }
I20240502 02:01:59.521997 31428 raft_consensus.cc:2932] T 3475ca93c949435699b590bd97e3735c P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7da2c0838fce40c5b5000c4d71c72708 (127.25.109.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: true } } }
I20240502 02:01:59.522572 31291 raft_consensus.cc:2932] T 3475ca93c949435699b590bd97e3735c P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7da2c0838fce40c5b5000c4d71c72708 (127.25.109.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: true } } }
W20240502 02:01:59.528306 31494 consensus_peers.cc:485] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 -> Peer 7da2c0838fce40c5b5000c4d71c72708 (127.25.109.133:36143): Couldn't send request to peer 7da2c0838fce40c5b5000c4d71c72708. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3475ca93c949435699b590bd97e3735c. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:01:59.528385 31144 catalog_manager.cc:5484] T 3475ca93c949435699b590bd97e3735c P 0db0855dad46416094c68fc878712cbb reported cstate change: config changed from index -1 to 4, NON_VOTER 7da2c0838fce40c5b5000c4d71c72708 (127.25.109.133) added. New cstate: current_term: 1 leader_uuid: "e325c0c7447f4b329994aafdee580666" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: true } } }
I20240502 02:01:59.540366 31428 consensus_queue.cc:237] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: true } }
I20240502 02:01:59.542510 31558 raft_consensus.cc:1261] T fecea55aa2ab478d91bcad93eca93aef P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Refusing update from remote peer 0db0855dad46416094c68fc878712cbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 02:01:59.542833 31360 consensus_peers.cc:485] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb -> Peer 29755ed6e9444b2daf5807bd23451d29 (127.25.109.132:38023): Couldn't send request to peer 29755ed6e9444b2daf5807bd23451d29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fecea55aa2ab478d91bcad93eca93aef. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:01:59.542907 31291 raft_consensus.cc:1261] T fecea55aa2ab478d91bcad93eca93aef P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Refusing update from remote peer 0db0855dad46416094c68fc878712cbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 02:01:59.543133 32576 consensus_queue.cc:1035] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:01:59.543462 32128 consensus_queue.cc:1035] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:01:59.546203 32572 raft_consensus.cc:2932] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 29755ed6e9444b2daf5807bd23451d29 (127.25.109.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: true } } }
I20240502 02:01:59.546952 31291 raft_consensus.cc:2932] T fecea55aa2ab478d91bcad93eca93aef P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 29755ed6e9444b2daf5807bd23451d29 (127.25.109.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: true } } }
I20240502 02:01:59.546962 31558 raft_consensus.cc:2932] T fecea55aa2ab478d91bcad93eca93aef P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 29755ed6e9444b2daf5807bd23451d29 (127.25.109.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: true } } }
I20240502 02:01:59.552594 31150 catalog_manager.cc:5484] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb reported cstate change: config changed from index -1 to 4, NON_VOTER 29755ed6e9444b2daf5807bd23451d29 (127.25.109.132) added. New cstate: current_term: 1 leader_uuid: "0db0855dad46416094c68fc878712cbb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 02:01:59.768237 32670 ts_tablet_manager.cc:902] T bec2e393d5e841c7857d6fd092aac53e P 7da2c0838fce40c5b5000c4d71c72708: Initiating tablet copy from peer 0db0855dad46416094c68fc878712cbb (127.25.109.130:38025)
I20240502 02:01:59.770016 32670 tablet_copy_client.cc:308] T bec2e393d5e841c7857d6fd092aac53e P 7da2c0838fce40c5b5000c4d71c72708: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.130:38025
I20240502 02:01:59.777133 31448 tablet_copy_service.cc:140] P 0db0855dad46416094c68fc878712cbb: Received BeginTabletCopySession request for tablet bec2e393d5e841c7857d6fd092aac53e from peer 7da2c0838fce40c5b5000c4d71c72708 ({username='slave'} at 127.25.109.133:55755)
I20240502 02:01:59.777395 31448 tablet_copy_service.cc:161] P 0db0855dad46416094c68fc878712cbb: Beginning new tablet copy session on tablet bec2e393d5e841c7857d6fd092aac53e from peer 7da2c0838fce40c5b5000c4d71c72708 at {username='slave'} at 127.25.109.133:55755: session id = 7da2c0838fce40c5b5000c4d71c72708-bec2e393d5e841c7857d6fd092aac53e
I20240502 02:01:59.780148 31448 tablet_copy_source_session.cc:205] T bec2e393d5e841c7857d6fd092aac53e P 0db0855dad46416094c68fc878712cbb: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 02:01:59.784945 32670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bec2e393d5e841c7857d6fd092aac53e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:59.795414 32670 tablet_copy_client.cc:783] T bec2e393d5e841c7857d6fd092aac53e P 7da2c0838fce40c5b5000c4d71c72708: tablet copy: Starting download of 0 data blocks...
I20240502 02:01:59.795768 32670 tablet_copy_client.cc:647] T bec2e393d5e841c7857d6fd092aac53e P 7da2c0838fce40c5b5000c4d71c72708: tablet copy: Starting download of 1 WAL segments...
I20240502 02:01:59.798276 32670 tablet_copy_client.cc:523] T bec2e393d5e841c7857d6fd092aac53e P 7da2c0838fce40c5b5000c4d71c72708: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:01:59.801680 32670 tablet_bootstrap.cc:492] T bec2e393d5e841c7857d6fd092aac53e P 7da2c0838fce40c5b5000c4d71c72708: Bootstrap starting.
I20240502 02:01:59.802608 32676 ts_tablet_manager.cc:902] T 1b97531b47d44bf8a7f905ef339f857a P 29755ed6e9444b2daf5807bd23451d29: Initiating tablet copy from peer 64688d2615e44408ab868cb531540a84 (127.25.109.129:33021)
I20240502 02:01:59.805137 32676 tablet_copy_client.cc:308] T 1b97531b47d44bf8a7f905ef339f857a P 29755ed6e9444b2daf5807bd23451d29: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.129:33021
I20240502 02:01:59.812701 31314 tablet_copy_service.cc:140] P 64688d2615e44408ab868cb531540a84: Received BeginTabletCopySession request for tablet 1b97531b47d44bf8a7f905ef339f857a from peer 29755ed6e9444b2daf5807bd23451d29 ({username='slave'} at 127.25.109.132:46187)
I20240502 02:01:59.813059 31314 tablet_copy_service.cc:161] P 64688d2615e44408ab868cb531540a84: Beginning new tablet copy session on tablet 1b97531b47d44bf8a7f905ef339f857a from peer 29755ed6e9444b2daf5807bd23451d29 at {username='slave'} at 127.25.109.132:46187: session id = 29755ed6e9444b2daf5807bd23451d29-1b97531b47d44bf8a7f905ef339f857a
I20240502 02:01:59.814424 32670 log.cc:824] T bec2e393d5e841c7857d6fd092aac53e P 7da2c0838fce40c5b5000c4d71c72708: Log is configured to *not* fsync() on all Append() calls
I20240502 02:01:59.816679 31314 tablet_copy_source_session.cc:205] T 1b97531b47d44bf8a7f905ef339f857a P 64688d2615e44408ab868cb531540a84: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 02:01:59.821768 32676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b97531b47d44bf8a7f905ef339f857a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:01:59.832386 32676 tablet_copy_client.cc:783] T 1b97531b47d44bf8a7f905ef339f857a P 29755ed6e9444b2daf5807bd23451d29: tablet copy: Starting download of 0 data blocks...
I20240502 02:01:59.832787 32676 tablet_copy_client.cc:647] T 1b97531b47d44bf8a7f905ef339f857a P 29755ed6e9444b2daf5807bd23451d29: tablet copy: Starting download of 1 WAL segments...
I20240502 02:01:59.835644 32676 tablet_copy_client.cc:523] T 1b97531b47d44bf8a7f905ef339f857a P 29755ed6e9444b2daf5807bd23451d29: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:01:59.840341 32676 tablet_bootstrap.cc:492] T 1b97531b47d44bf8a7f905ef339f857a P 29755ed6e9444b2daf5807bd23451d29: Bootstrap starting.
I20240502 02:01:59.853982 32670 tablet_bootstrap.cc:492] T bec2e393d5e841c7857d6fd092aac53e P 7da2c0838fce40c5b5000c4d71c72708: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:01:59.853935 32676 log.cc:824] T 1b97531b47d44bf8a7f905ef339f857a P 29755ed6e9444b2daf5807bd23451d29: Log is configured to *not* fsync() on all Append() calls
I20240502 02:01:59.854758 32670 tablet_bootstrap.cc:492] T bec2e393d5e841c7857d6fd092aac53e P 7da2c0838fce40c5b5000c4d71c72708: Bootstrap complete.
I20240502 02:01:59.855319 32670 ts_tablet_manager.cc:1367] T bec2e393d5e841c7857d6fd092aac53e P 7da2c0838fce40c5b5000c4d71c72708: Time spent bootstrapping tablet: real 0.054s	user 0.044s	sys 0.010s
I20240502 02:01:59.861788 32670 raft_consensus.cc:348] T bec2e393d5e841c7857d6fd092aac53e P 7da2c0838fce40c5b5000c4d71c72708 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: true } }
I20240502 02:01:59.862344 32670 raft_consensus.cc:729] T bec2e393d5e841c7857d6fd092aac53e P 7da2c0838fce40c5b5000c4d71c72708 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7da2c0838fce40c5b5000c4d71c72708, State: Initialized, Role: LEARNER
I20240502 02:01:59.862973 32670 consensus_queue.cc:260] T bec2e393d5e841c7857d6fd092aac53e P 7da2c0838fce40c5b5000c4d71c72708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: true } }
I20240502 02:01:59.870414 32670 ts_tablet_manager.cc:1398] T bec2e393d5e841c7857d6fd092aac53e P 7da2c0838fce40c5b5000c4d71c72708: Time spent starting tablet: real 0.015s	user 0.010s	sys 0.003s
I20240502 02:01:59.871672 31448 tablet_copy_service.cc:342] P 0db0855dad46416094c68fc878712cbb: Request end of tablet copy session 7da2c0838fce40c5b5000c4d71c72708-bec2e393d5e841c7857d6fd092aac53e received from {username='slave'} at 127.25.109.133:55755
I20240502 02:01:59.871928 31448 tablet_copy_service.cc:433] P 0db0855dad46416094c68fc878712cbb: ending tablet copy session 7da2c0838fce40c5b5000c4d71c72708-bec2e393d5e841c7857d6fd092aac53e on tablet bec2e393d5e841c7857d6fd092aac53e with peer 7da2c0838fce40c5b5000c4d71c72708
I20240502 02:01:59.899349 32676 tablet_bootstrap.cc:492] T 1b97531b47d44bf8a7f905ef339f857a P 29755ed6e9444b2daf5807bd23451d29: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:01:59.899984 32676 tablet_bootstrap.cc:492] T 1b97531b47d44bf8a7f905ef339f857a P 29755ed6e9444b2daf5807bd23451d29: Bootstrap complete.
I20240502 02:01:59.900537 32676 ts_tablet_manager.cc:1367] T 1b97531b47d44bf8a7f905ef339f857a P 29755ed6e9444b2daf5807bd23451d29: Time spent bootstrapping tablet: real 0.060s	user 0.053s	sys 0.004s
I20240502 02:01:59.906435 32676 raft_consensus.cc:348] T 1b97531b47d44bf8a7f905ef339f857a P 29755ed6e9444b2daf5807bd23451d29 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: true } }
I20240502 02:01:59.906873 32676 raft_consensus.cc:729] T 1b97531b47d44bf8a7f905ef339f857a P 29755ed6e9444b2daf5807bd23451d29 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 29755ed6e9444b2daf5807bd23451d29, State: Initialized, Role: LEARNER
I20240502 02:01:59.907414 32676 consensus_queue.cc:260] T 1b97531b47d44bf8a7f905ef339f857a P 29755ed6e9444b2daf5807bd23451d29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: true } }
I20240502 02:01:59.909216 32676 ts_tablet_manager.cc:1398] T 1b97531b47d44bf8a7f905ef339f857a P 29755ed6e9444b2daf5807bd23451d29: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20240502 02:01:59.910341 31314 tablet_copy_service.cc:342] P 64688d2615e44408ab868cb531540a84: Request end of tablet copy session 29755ed6e9444b2daf5807bd23451d29-1b97531b47d44bf8a7f905ef339f857a received from {username='slave'} at 127.25.109.132:46187
I20240502 02:01:59.910574 31314 tablet_copy_service.cc:433] P 64688d2615e44408ab868cb531540a84: ending tablet copy session 29755ed6e9444b2daf5807bd23451d29-1b97531b47d44bf8a7f905ef339f857a on tablet 1b97531b47d44bf8a7f905ef339f857a with peer 29755ed6e9444b2daf5807bd23451d29
I20240502 02:01:59.996415 32670 ts_tablet_manager.cc:902] T 171f811ca5584cd3ab08fc25a9b7f7fc P 7da2c0838fce40c5b5000c4d71c72708: Initiating tablet copy from peer 64688d2615e44408ab868cb531540a84 (127.25.109.129:33021)
I20240502 02:01:59.997601 32670 tablet_copy_client.cc:308] T 171f811ca5584cd3ab08fc25a9b7f7fc P 7da2c0838fce40c5b5000c4d71c72708: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.129:33021
I20240502 02:02:00.000579 32676 ts_tablet_manager.cc:902] T a10fd1feee3a4838a8fba1599a1b890d P 29755ed6e9444b2daf5807bd23451d29: Initiating tablet copy from peer 64688d2615e44408ab868cb531540a84 (127.25.109.129:33021)
I20240502 02:02:00.001752 32676 tablet_copy_client.cc:308] T a10fd1feee3a4838a8fba1599a1b890d P 29755ed6e9444b2daf5807bd23451d29: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.129:33021
I20240502 02:02:00.002463 31314 tablet_copy_service.cc:140] P 64688d2615e44408ab868cb531540a84: Received BeginTabletCopySession request for tablet a10fd1feee3a4838a8fba1599a1b890d from peer 29755ed6e9444b2daf5807bd23451d29 ({username='slave'} at 127.25.109.132:46187)
I20240502 02:02:00.002684 31314 tablet_copy_service.cc:161] P 64688d2615e44408ab868cb531540a84: Beginning new tablet copy session on tablet a10fd1feee3a4838a8fba1599a1b890d from peer 29755ed6e9444b2daf5807bd23451d29 at {username='slave'} at 127.25.109.132:46187: session id = 29755ed6e9444b2daf5807bd23451d29-a10fd1feee3a4838a8fba1599a1b890d
I20240502 02:02:00.003971 31313 tablet_copy_service.cc:140] P 64688d2615e44408ab868cb531540a84: Received BeginTabletCopySession request for tablet 171f811ca5584cd3ab08fc25a9b7f7fc from peer 7da2c0838fce40c5b5000c4d71c72708 ({username='slave'} at 127.25.109.133:43105)
I20240502 02:02:00.004113 31313 tablet_copy_service.cc:161] P 64688d2615e44408ab868cb531540a84: Beginning new tablet copy session on tablet 171f811ca5584cd3ab08fc25a9b7f7fc from peer 7da2c0838fce40c5b5000c4d71c72708 at {username='slave'} at 127.25.109.133:43105: session id = 7da2c0838fce40c5b5000c4d71c72708-171f811ca5584cd3ab08fc25a9b7f7fc
I20240502 02:02:00.005014 31314 tablet_copy_source_session.cc:205] T a10fd1feee3a4838a8fba1599a1b890d P 64688d2615e44408ab868cb531540a84: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 02:02:00.005985 32735 ts_tablet_manager.cc:902] T 28de9be2be73409387480db617b0031c P 29755ed6e9444b2daf5807bd23451d29: Initiating tablet copy from peer 0db0855dad46416094c68fc878712cbb (127.25.109.130:38025)
I20240502 02:02:00.006392 32676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a10fd1feee3a4838a8fba1599a1b890d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:00.006717 31313 tablet_copy_source_session.cc:205] T 171f811ca5584cd3ab08fc25a9b7f7fc P 64688d2615e44408ab868cb531540a84: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 02:02:00.008139 32670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 171f811ca5584cd3ab08fc25a9b7f7fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:00.011783 32676 tablet_copy_client.cc:783] T a10fd1feee3a4838a8fba1599a1b890d P 29755ed6e9444b2daf5807bd23451d29: tablet copy: Starting download of 0 data blocks...
I20240502 02:02:00.012154 32676 tablet_copy_client.cc:647] T a10fd1feee3a4838a8fba1599a1b890d P 29755ed6e9444b2daf5807bd23451d29: tablet copy: Starting download of 1 WAL segments...
I20240502 02:02:00.013875 32735 tablet_copy_client.cc:308] T 28de9be2be73409387480db617b0031c P 29755ed6e9444b2daf5807bd23451d29: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.130:38025
I20240502 02:02:00.014555 32676 tablet_copy_client.cc:523] T a10fd1feee3a4838a8fba1599a1b890d P 29755ed6e9444b2daf5807bd23451d29: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:02:00.014770 32670 tablet_copy_client.cc:783] T 171f811ca5584cd3ab08fc25a9b7f7fc P 7da2c0838fce40c5b5000c4d71c72708: tablet copy: Starting download of 0 data blocks...
I20240502 02:02:00.014986 32670 tablet_copy_client.cc:647] T 171f811ca5584cd3ab08fc25a9b7f7fc P 7da2c0838fce40c5b5000c4d71c72708: tablet copy: Starting download of 1 WAL segments...
I20240502 02:02:00.016700 32670 tablet_copy_client.cc:523] T 171f811ca5584cd3ab08fc25a9b7f7fc P 7da2c0838fce40c5b5000c4d71c72708: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:02:00.018033 32676 tablet_bootstrap.cc:492] T a10fd1feee3a4838a8fba1599a1b890d P 29755ed6e9444b2daf5807bd23451d29: Bootstrap starting.
I20240502 02:02:00.020273 32670 tablet_bootstrap.cc:492] T 171f811ca5584cd3ab08fc25a9b7f7fc P 7da2c0838fce40c5b5000c4d71c72708: Bootstrap starting.
I20240502 02:02:00.023037 31448 tablet_copy_service.cc:140] P 0db0855dad46416094c68fc878712cbb: Received BeginTabletCopySession request for tablet 28de9be2be73409387480db617b0031c from peer 29755ed6e9444b2daf5807bd23451d29 ({username='slave'} at 127.25.109.132:44461)
I20240502 02:02:00.023236 31448 tablet_copy_service.cc:161] P 0db0855dad46416094c68fc878712cbb: Beginning new tablet copy session on tablet 28de9be2be73409387480db617b0031c from peer 29755ed6e9444b2daf5807bd23451d29 at {username='slave'} at 127.25.109.132:44461: session id = 29755ed6e9444b2daf5807bd23451d29-28de9be2be73409387480db617b0031c
I20240502 02:02:00.026160 31448 tablet_copy_source_session.cc:205] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 02:02:00.028061 32735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28de9be2be73409387480db617b0031c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:00.035645 32735 tablet_copy_client.cc:783] T 28de9be2be73409387480db617b0031c P 29755ed6e9444b2daf5807bd23451d29: tablet copy: Starting download of 0 data blocks...
I20240502 02:02:00.036027 32735 tablet_copy_client.cc:647] T 28de9be2be73409387480db617b0031c P 29755ed6e9444b2daf5807bd23451d29: tablet copy: Starting download of 1 WAL segments...
I20240502 02:02:00.038185 32735 tablet_copy_client.cc:523] T 28de9be2be73409387480db617b0031c P 29755ed6e9444b2daf5807bd23451d29: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:02:00.042001 32735 tablet_bootstrap.cc:492] T 28de9be2be73409387480db617b0031c P 29755ed6e9444b2daf5807bd23451d29: Bootstrap starting.
I20240502 02:02:00.068470 32743 ts_tablet_manager.cc:902] T 3475ca93c949435699b590bd97e3735c P 7da2c0838fce40c5b5000c4d71c72708: Initiating tablet copy from peer e325c0c7447f4b329994aafdee580666 (127.25.109.131:43937)
I20240502 02:02:00.076231 32676 tablet_bootstrap.cc:492] T a10fd1feee3a4838a8fba1599a1b890d P 29755ed6e9444b2daf5807bd23451d29: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:02:00.076822 32676 tablet_bootstrap.cc:492] T a10fd1feee3a4838a8fba1599a1b890d P 29755ed6e9444b2daf5807bd23451d29: Bootstrap complete.
I20240502 02:02:00.077227 32676 ts_tablet_manager.cc:1367] T a10fd1feee3a4838a8fba1599a1b890d P 29755ed6e9444b2daf5807bd23451d29: Time spent bootstrapping tablet: real 0.059s	user 0.050s	sys 0.008s
I20240502 02:02:00.078469 32676 raft_consensus.cc:348] T a10fd1feee3a4838a8fba1599a1b890d P 29755ed6e9444b2daf5807bd23451d29 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: true } }
I20240502 02:02:00.078855 32676 raft_consensus.cc:729] T a10fd1feee3a4838a8fba1599a1b890d P 29755ed6e9444b2daf5807bd23451d29 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 29755ed6e9444b2daf5807bd23451d29, State: Initialized, Role: LEARNER
I20240502 02:02:00.079136 32676 consensus_queue.cc:260] T a10fd1feee3a4838a8fba1599a1b890d P 29755ed6e9444b2daf5807bd23451d29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: true } }
I20240502 02:02:00.079818 32676 ts_tablet_manager.cc:1398] T a10fd1feee3a4838a8fba1599a1b890d P 29755ed6e9444b2daf5807bd23451d29: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 02:02:00.083050 31313 tablet_copy_service.cc:342] P 64688d2615e44408ab868cb531540a84: Request end of tablet copy session 29755ed6e9444b2daf5807bd23451d29-a10fd1feee3a4838a8fba1599a1b890d received from {username='slave'} at 127.25.109.132:46187
I20240502 02:02:00.083201 31313 tablet_copy_service.cc:433] P 64688d2615e44408ab868cb531540a84: ending tablet copy session 29755ed6e9444b2daf5807bd23451d29-a10fd1feee3a4838a8fba1599a1b890d on tablet a10fd1feee3a4838a8fba1599a1b890d with peer 29755ed6e9444b2daf5807bd23451d29
I20240502 02:02:00.086607 32670 tablet_bootstrap.cc:492] T 171f811ca5584cd3ab08fc25a9b7f7fc P 7da2c0838fce40c5b5000c4d71c72708: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:02:00.086983 32670 tablet_bootstrap.cc:492] T 171f811ca5584cd3ab08fc25a9b7f7fc P 7da2c0838fce40c5b5000c4d71c72708: Bootstrap complete.
I20240502 02:02:00.087247 32670 ts_tablet_manager.cc:1367] T 171f811ca5584cd3ab08fc25a9b7f7fc P 7da2c0838fce40c5b5000c4d71c72708: Time spent bootstrapping tablet: real 0.067s	user 0.056s	sys 0.004s
I20240502 02:02:00.088210 32670 raft_consensus.cc:348] T 171f811ca5584cd3ab08fc25a9b7f7fc P 7da2c0838fce40c5b5000c4d71c72708 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: true } }
I20240502 02:02:00.088611 32670 raft_consensus.cc:729] T 171f811ca5584cd3ab08fc25a9b7f7fc P 7da2c0838fce40c5b5000c4d71c72708 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7da2c0838fce40c5b5000c4d71c72708, State: Initialized, Role: LEARNER
I20240502 02:02:00.088956 32670 consensus_queue.cc:260] T 171f811ca5584cd3ab08fc25a9b7f7fc P 7da2c0838fce40c5b5000c4d71c72708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: true } }
I20240502 02:02:00.092624 32670 ts_tablet_manager.cc:1398] T 171f811ca5584cd3ab08fc25a9b7f7fc P 7da2c0838fce40c5b5000c4d71c72708: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.000s
I20240502 02:02:00.096160 31313 tablet_copy_service.cc:342] P 64688d2615e44408ab868cb531540a84: Request end of tablet copy session 7da2c0838fce40c5b5000c4d71c72708-171f811ca5584cd3ab08fc25a9b7f7fc received from {username='slave'} at 127.25.109.133:43105
I20240502 02:02:00.096340 31313 tablet_copy_service.cc:433] P 64688d2615e44408ab868cb531540a84: ending tablet copy session 7da2c0838fce40c5b5000c4d71c72708-171f811ca5584cd3ab08fc25a9b7f7fc on tablet 171f811ca5584cd3ab08fc25a9b7f7fc with peer 7da2c0838fce40c5b5000c4d71c72708
I20240502 02:02:00.098800 32743 tablet_copy_client.cc:308] T 3475ca93c949435699b590bd97e3735c P 7da2c0838fce40c5b5000c4d71c72708: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.131:43937
I20240502 02:02:00.100507 32735 tablet_bootstrap.cc:492] T 28de9be2be73409387480db617b0031c P 29755ed6e9444b2daf5807bd23451d29: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:02:00.100972 32735 tablet_bootstrap.cc:492] T 28de9be2be73409387480db617b0031c P 29755ed6e9444b2daf5807bd23451d29: Bootstrap complete.
I20240502 02:02:00.101269 32735 ts_tablet_manager.cc:1367] T 28de9be2be73409387480db617b0031c P 29755ed6e9444b2daf5807bd23451d29: Time spent bootstrapping tablet: real 0.059s	user 0.039s	sys 0.011s
I20240502 02:02:00.102077 32735 raft_consensus.cc:348] T 28de9be2be73409387480db617b0031c P 29755ed6e9444b2daf5807bd23451d29 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: true } }
I20240502 02:02:00.102394 32735 raft_consensus.cc:729] T 28de9be2be73409387480db617b0031c P 29755ed6e9444b2daf5807bd23451d29 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 29755ed6e9444b2daf5807bd23451d29, State: Initialized, Role: LEARNER
I20240502 02:02:00.102654 32735 consensus_queue.cc:260] T 28de9be2be73409387480db617b0031c P 29755ed6e9444b2daf5807bd23451d29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: true } }
I20240502 02:02:00.103997 32735 ts_tablet_manager.cc:1398] T 28de9be2be73409387480db617b0031c P 29755ed6e9444b2daf5807bd23451d29: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 02:02:00.104936 31581 tablet_copy_service.cc:140] P e325c0c7447f4b329994aafdee580666: Received BeginTabletCopySession request for tablet 3475ca93c949435699b590bd97e3735c from peer 7da2c0838fce40c5b5000c4d71c72708 ({username='slave'} at 127.25.109.133:58369)
I20240502 02:02:00.105198 31581 tablet_copy_service.cc:161] P e325c0c7447f4b329994aafdee580666: Beginning new tablet copy session on tablet 3475ca93c949435699b590bd97e3735c from peer 7da2c0838fce40c5b5000c4d71c72708 at {username='slave'} at 127.25.109.133:58369: session id = 7da2c0838fce40c5b5000c4d71c72708-3475ca93c949435699b590bd97e3735c
I20240502 02:02:00.105990 31448 tablet_copy_service.cc:342] P 0db0855dad46416094c68fc878712cbb: Request end of tablet copy session 29755ed6e9444b2daf5807bd23451d29-28de9be2be73409387480db617b0031c received from {username='slave'} at 127.25.109.132:44461
I20240502 02:02:00.106142 31448 tablet_copy_service.cc:433] P 0db0855dad46416094c68fc878712cbb: ending tablet copy session 29755ed6e9444b2daf5807bd23451d29-28de9be2be73409387480db617b0031c on tablet 28de9be2be73409387480db617b0031c with peer 29755ed6e9444b2daf5807bd23451d29
I20240502 02:02:00.106317 32670 ts_tablet_manager.cc:902] T 9676b44f15e84ae28a64caa71f5722c6 P 7da2c0838fce40c5b5000c4d71c72708: Initiating tablet copy from peer 64688d2615e44408ab868cb531540a84 (127.25.109.129:33021)
I20240502 02:02:00.109370 31581 tablet_copy_source_session.cc:205] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 02:02:00.112243 32743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3475ca93c949435699b590bd97e3735c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:00.116818 32670 tablet_copy_client.cc:308] T 9676b44f15e84ae28a64caa71f5722c6 P 7da2c0838fce40c5b5000c4d71c72708: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.129:33021
I20240502 02:02:00.117700 31313 tablet_copy_service.cc:140] P 64688d2615e44408ab868cb531540a84: Received BeginTabletCopySession request for tablet 9676b44f15e84ae28a64caa71f5722c6 from peer 7da2c0838fce40c5b5000c4d71c72708 ({username='slave'} at 127.25.109.133:43105)
I20240502 02:02:00.117887 31313 tablet_copy_service.cc:161] P 64688d2615e44408ab868cb531540a84: Beginning new tablet copy session on tablet 9676b44f15e84ae28a64caa71f5722c6 from peer 7da2c0838fce40c5b5000c4d71c72708 at {username='slave'} at 127.25.109.133:43105: session id = 7da2c0838fce40c5b5000c4d71c72708-9676b44f15e84ae28a64caa71f5722c6
I20240502 02:02:00.119961 31313 tablet_copy_source_session.cc:205] T 9676b44f15e84ae28a64caa71f5722c6 P 64688d2615e44408ab868cb531540a84: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 02:02:00.120045 32743 tablet_copy_client.cc:783] T 3475ca93c949435699b590bd97e3735c P 7da2c0838fce40c5b5000c4d71c72708: tablet copy: Starting download of 0 data blocks...
I20240502 02:02:00.120276 32743 tablet_copy_client.cc:647] T 3475ca93c949435699b590bd97e3735c P 7da2c0838fce40c5b5000c4d71c72708: tablet copy: Starting download of 1 WAL segments...
I20240502 02:02:00.121147 32670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9676b44f15e84ae28a64caa71f5722c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:00.122612 32743 tablet_copy_client.cc:523] T 3475ca93c949435699b590bd97e3735c P 7da2c0838fce40c5b5000c4d71c72708: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:02:00.125874 32670 tablet_copy_client.cc:783] T 9676b44f15e84ae28a64caa71f5722c6 P 7da2c0838fce40c5b5000c4d71c72708: tablet copy: Starting download of 0 data blocks...
I20240502 02:02:00.126135 32670 tablet_copy_client.cc:647] T 9676b44f15e84ae28a64caa71f5722c6 P 7da2c0838fce40c5b5000c4d71c72708: tablet copy: Starting download of 1 WAL segments...
I20240502 02:02:00.126706 32743 tablet_bootstrap.cc:492] T 3475ca93c949435699b590bd97e3735c P 7da2c0838fce40c5b5000c4d71c72708: Bootstrap starting.
I20240502 02:02:00.127822 32670 tablet_copy_client.cc:523] T 9676b44f15e84ae28a64caa71f5722c6 P 7da2c0838fce40c5b5000c4d71c72708: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:02:00.130293 32670 tablet_bootstrap.cc:492] T 9676b44f15e84ae28a64caa71f5722c6 P 7da2c0838fce40c5b5000c4d71c72708: Bootstrap starting.
I20240502 02:02:00.133813 32735 ts_tablet_manager.cc:902] T fecea55aa2ab478d91bcad93eca93aef P 29755ed6e9444b2daf5807bd23451d29: Initiating tablet copy from peer 0db0855dad46416094c68fc878712cbb (127.25.109.130:38025)
I20240502 02:02:00.173173 32735 tablet_copy_client.cc:308] T fecea55aa2ab478d91bcad93eca93aef P 29755ed6e9444b2daf5807bd23451d29: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.130:38025
I20240502 02:02:00.174158 31448 tablet_copy_service.cc:140] P 0db0855dad46416094c68fc878712cbb: Received BeginTabletCopySession request for tablet fecea55aa2ab478d91bcad93eca93aef from peer 29755ed6e9444b2daf5807bd23451d29 ({username='slave'} at 127.25.109.132:44461)
I20240502 02:02:00.174343 31448 tablet_copy_service.cc:161] P 0db0855dad46416094c68fc878712cbb: Beginning new tablet copy session on tablet fecea55aa2ab478d91bcad93eca93aef from peer 29755ed6e9444b2daf5807bd23451d29 at {username='slave'} at 127.25.109.132:44461: session id = 29755ed6e9444b2daf5807bd23451d29-fecea55aa2ab478d91bcad93eca93aef
I20240502 02:02:00.177450 31448 tablet_copy_source_session.cc:205] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 02:02:00.180105 32735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fecea55aa2ab478d91bcad93eca93aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:00.185245 32670 tablet_bootstrap.cc:492] T 9676b44f15e84ae28a64caa71f5722c6 P 7da2c0838fce40c5b5000c4d71c72708: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:02:00.185846 32670 tablet_bootstrap.cc:492] T 9676b44f15e84ae28a64caa71f5722c6 P 7da2c0838fce40c5b5000c4d71c72708: Bootstrap complete.
I20240502 02:02:00.186196 32670 ts_tablet_manager.cc:1367] T 9676b44f15e84ae28a64caa71f5722c6 P 7da2c0838fce40c5b5000c4d71c72708: Time spent bootstrapping tablet: real 0.056s	user 0.049s	sys 0.007s
I20240502 02:02:00.187134 32735 tablet_copy_client.cc:783] T fecea55aa2ab478d91bcad93eca93aef P 29755ed6e9444b2daf5807bd23451d29: tablet copy: Starting download of 0 data blocks...
I20240502 02:02:00.187172 32670 raft_consensus.cc:348] T 9676b44f15e84ae28a64caa71f5722c6 P 7da2c0838fce40c5b5000c4d71c72708 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: true } }
I20240502 02:02:00.187436 32735 tablet_copy_client.cc:647] T fecea55aa2ab478d91bcad93eca93aef P 29755ed6e9444b2daf5807bd23451d29: tablet copy: Starting download of 1 WAL segments...
I20240502 02:02:00.187417 32743 tablet_bootstrap.cc:492] T 3475ca93c949435699b590bd97e3735c P 7da2c0838fce40c5b5000c4d71c72708: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:02:00.187541 32670 raft_consensus.cc:729] T 9676b44f15e84ae28a64caa71f5722c6 P 7da2c0838fce40c5b5000c4d71c72708 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7da2c0838fce40c5b5000c4d71c72708, State: Initialized, Role: LEARNER
I20240502 02:02:00.188136 32743 tablet_bootstrap.cc:492] T 3475ca93c949435699b590bd97e3735c P 7da2c0838fce40c5b5000c4d71c72708: Bootstrap complete.
I20240502 02:02:00.187968 32670 consensus_queue.cc:260] T 9676b44f15e84ae28a64caa71f5722c6 P 7da2c0838fce40c5b5000c4d71c72708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: true } }
I20240502 02:02:00.188519 32743 ts_tablet_manager.cc:1367] T 3475ca93c949435699b590bd97e3735c P 7da2c0838fce40c5b5000c4d71c72708: Time spent bootstrapping tablet: real 0.062s	user 0.049s	sys 0.012s
I20240502 02:02:00.189345 32670 ts_tablet_manager.cc:1398] T 9676b44f15e84ae28a64caa71f5722c6 P 7da2c0838fce40c5b5000c4d71c72708: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 02:02:00.189363 32735 tablet_copy_client.cc:523] T fecea55aa2ab478d91bcad93eca93aef P 29755ed6e9444b2daf5807bd23451d29: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:02:00.189963 32743 raft_consensus.cc:348] T 3475ca93c949435699b590bd97e3735c P 7da2c0838fce40c5b5000c4d71c72708 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: true } }
I20240502 02:02:00.190285 32743 raft_consensus.cc:729] T 3475ca93c949435699b590bd97e3735c P 7da2c0838fce40c5b5000c4d71c72708 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7da2c0838fce40c5b5000c4d71c72708, State: Initialized, Role: LEARNER
I20240502 02:02:00.190655 31313 tablet_copy_service.cc:342] P 64688d2615e44408ab868cb531540a84: Request end of tablet copy session 7da2c0838fce40c5b5000c4d71c72708-9676b44f15e84ae28a64caa71f5722c6 received from {username='slave'} at 127.25.109.133:43105
I20240502 02:02:00.190538 32743 consensus_queue.cc:260] T 3475ca93c949435699b590bd97e3735c P 7da2c0838fce40c5b5000c4d71c72708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: NON_VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: true } }
I20240502 02:02:00.190791 31313 tablet_copy_service.cc:433] P 64688d2615e44408ab868cb531540a84: ending tablet copy session 7da2c0838fce40c5b5000c4d71c72708-9676b44f15e84ae28a64caa71f5722c6 on tablet 9676b44f15e84ae28a64caa71f5722c6 with peer 7da2c0838fce40c5b5000c4d71c72708
I20240502 02:02:00.191232 32743 ts_tablet_manager.cc:1398] T 3475ca93c949435699b590bd97e3735c P 7da2c0838fce40c5b5000c4d71c72708: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240502 02:02:00.192363 31581 tablet_copy_service.cc:342] P e325c0c7447f4b329994aafdee580666: Request end of tablet copy session 7da2c0838fce40c5b5000c4d71c72708-3475ca93c949435699b590bd97e3735c received from {username='slave'} at 127.25.109.133:58369
I20240502 02:02:00.192566 31581 tablet_copy_service.cc:433] P e325c0c7447f4b329994aafdee580666: ending tablet copy session 7da2c0838fce40c5b5000c4d71c72708-3475ca93c949435699b590bd97e3735c on tablet 3475ca93c949435699b590bd97e3735c with peer 7da2c0838fce40c5b5000c4d71c72708
I20240502 02:02:00.192772 32735 tablet_bootstrap.cc:492] T fecea55aa2ab478d91bcad93eca93aef P 29755ed6e9444b2daf5807bd23451d29: Bootstrap starting.
I20240502 02:02:00.227308 32735 tablet_bootstrap.cc:492] T fecea55aa2ab478d91bcad93eca93aef P 29755ed6e9444b2daf5807bd23451d29: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:02:00.227820 32735 tablet_bootstrap.cc:492] T fecea55aa2ab478d91bcad93eca93aef P 29755ed6e9444b2daf5807bd23451d29: Bootstrap complete.
I20240502 02:02:00.228247 32735 ts_tablet_manager.cc:1367] T fecea55aa2ab478d91bcad93eca93aef P 29755ed6e9444b2daf5807bd23451d29: Time spent bootstrapping tablet: real 0.036s	user 0.031s	sys 0.005s
I20240502 02:02:00.229142 32735 raft_consensus.cc:348] T fecea55aa2ab478d91bcad93eca93aef P 29755ed6e9444b2daf5807bd23451d29 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: true } }
I20240502 02:02:00.229386 32735 raft_consensus.cc:729] T fecea55aa2ab478d91bcad93eca93aef P 29755ed6e9444b2daf5807bd23451d29 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 29755ed6e9444b2daf5807bd23451d29, State: Initialized, Role: LEARNER
I20240502 02:02:00.229625 32735 consensus_queue.cc:260] T fecea55aa2ab478d91bcad93eca93aef P 29755ed6e9444b2daf5807bd23451d29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: true } }
I20240502 02:02:00.230166 32735 ts_tablet_manager.cc:1398] T fecea55aa2ab478d91bcad93eca93aef P 29755ed6e9444b2daf5807bd23451d29: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 02:02:00.231197 31448 tablet_copy_service.cc:342] P 0db0855dad46416094c68fc878712cbb: Request end of tablet copy session 29755ed6e9444b2daf5807bd23451d29-fecea55aa2ab478d91bcad93eca93aef received from {username='slave'} at 127.25.109.132:44461
I20240502 02:02:00.231385 31448 tablet_copy_service.cc:433] P 0db0855dad46416094c68fc878712cbb: ending tablet copy session 29755ed6e9444b2daf5807bd23451d29-fecea55aa2ab478d91bcad93eca93aef on tablet fecea55aa2ab478d91bcad93eca93aef with peer 29755ed6e9444b2daf5807bd23451d29
I20240502 02:02:00.246234 32364 raft_consensus.cc:1203] T bec2e393d5e841c7857d6fd092aac53e P 7da2c0838fce40c5b5000c4d71c72708 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 02:02:00.388451 32230 raft_consensus.cc:1203] T 1b97531b47d44bf8a7f905ef339f857a P 29755ed6e9444b2daf5807bd23451d29 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 02:02:00.456482 32364 raft_consensus.cc:1203] T 3475ca93c949435699b590bd97e3735c P 7da2c0838fce40c5b5000c4d71c72708 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 02:02:00.467475 32230 raft_consensus.cc:1203] T a10fd1feee3a4838a8fba1599a1b890d P 29755ed6e9444b2daf5807bd23451d29 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 02:02:00.548058 32364 raft_consensus.cc:1203] T 9676b44f15e84ae28a64caa71f5722c6 P 7da2c0838fce40c5b5000c4d71c72708 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 02:02:00.578541 32364 raft_consensus.cc:1203] T 171f811ca5584cd3ab08fc25a9b7f7fc P 7da2c0838fce40c5b5000c4d71c72708 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 02:02:00.597990 32230 raft_consensus.cc:1203] T 28de9be2be73409387480db617b0031c P 29755ed6e9444b2daf5807bd23451d29 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 02:02:00.735983 32230 raft_consensus.cc:1203] T fecea55aa2ab478d91bcad93eca93aef P 29755ed6e9444b2daf5807bd23451d29 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 02:02:00.823143 32572 raft_consensus.cc:1050] T bec2e393d5e841c7857d6fd092aac53e P 0db0855dad46416094c68fc878712cbb: attempting to promote NON_VOTER 7da2c0838fce40c5b5000c4d71c72708 to VOTER
I20240502 02:02:00.824129 32572 consensus_queue.cc:237] T bec2e393d5e841c7857d6fd092aac53e P 0db0855dad46416094c68fc878712cbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } }
I20240502 02:02:00.826720 31292 raft_consensus.cc:1261] T bec2e393d5e841c7857d6fd092aac53e P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Refusing update from remote peer 0db0855dad46416094c68fc878712cbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:02:00.826778 31558 raft_consensus.cc:1261] T bec2e393d5e841c7857d6fd092aac53e P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Refusing update from remote peer 0db0855dad46416094c68fc878712cbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:02:00.826825 32364 raft_consensus.cc:1261] T bec2e393d5e841c7857d6fd092aac53e P 7da2c0838fce40c5b5000c4d71c72708 [term 1 LEARNER]: Refusing update from remote peer 0db0855dad46416094c68fc878712cbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:02:00.827456   419 consensus_queue.cc:1035] T bec2e393d5e841c7857d6fd092aac53e P 0db0855dad46416094c68fc878712cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:00.827898   324 consensus_queue.cc:1035] T bec2e393d5e841c7857d6fd092aac53e P 0db0855dad46416094c68fc878712cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:00.828200 32572 consensus_queue.cc:1035] T bec2e393d5e841c7857d6fd092aac53e P 0db0855dad46416094c68fc878712cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:00.832504 32572 raft_consensus.cc:2932] T bec2e393d5e841c7857d6fd092aac53e P 0db0855dad46416094c68fc878712cbb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7da2c0838fce40c5b5000c4d71c72708 (127.25.109.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } }
I20240502 02:02:00.833230 31558 raft_consensus.cc:2932] T bec2e393d5e841c7857d6fd092aac53e P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7da2c0838fce40c5b5000c4d71c72708 (127.25.109.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } }
I20240502 02:02:00.833361 32364 raft_consensus.cc:2932] T bec2e393d5e841c7857d6fd092aac53e P 7da2c0838fce40c5b5000c4d71c72708 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7da2c0838fce40c5b5000c4d71c72708 (127.25.109.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } }
I20240502 02:02:00.837926 31292 raft_consensus.cc:2932] T bec2e393d5e841c7857d6fd092aac53e P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7da2c0838fce40c5b5000c4d71c72708 (127.25.109.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } }
I20240502 02:02:00.838907 31144 catalog_manager.cc:5484] T bec2e393d5e841c7857d6fd092aac53e P 7da2c0838fce40c5b5000c4d71c72708 reported cstate change: config changed from index 4 to 5, 7da2c0838fce40c5b5000c4d71c72708 (127.25.109.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0db0855dad46416094c68fc878712cbb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } }
I20240502 02:02:00.846105 31428 consensus_queue.cc:237] T bec2e393d5e841c7857d6fd092aac53e P 0db0855dad46416094c68fc878712cbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } }
I20240502 02:02:00.848431 32364 raft_consensus.cc:1261] T bec2e393d5e841c7857d6fd092aac53e P 7da2c0838fce40c5b5000c4d71c72708 [term 1 FOLLOWER]: Refusing update from remote peer 0db0855dad46416094c68fc878712cbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:02:00.849136 32572 consensus_queue.cc:1035] T bec2e393d5e841c7857d6fd092aac53e P 0db0855dad46416094c68fc878712cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:02:00.849278 31558 raft_consensus.cc:1261] T bec2e393d5e841c7857d6fd092aac53e P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Refusing update from remote peer 0db0855dad46416094c68fc878712cbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:02:00.849812   323 consensus_queue.cc:1035] T bec2e393d5e841c7857d6fd092aac53e P 0db0855dad46416094c68fc878712cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:02:00.852041   419 raft_consensus.cc:2932] T bec2e393d5e841c7857d6fd092aac53e P 0db0855dad46416094c68fc878712cbb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 64688d2615e44408ab868cb531540a84 (127.25.109.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } }
I20240502 02:02:00.852696 31558 raft_consensus.cc:2932] T bec2e393d5e841c7857d6fd092aac53e P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 64688d2615e44408ab868cb531540a84 (127.25.109.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } }
I20240502 02:02:00.852792 32364 raft_consensus.cc:2932] T bec2e393d5e841c7857d6fd092aac53e P 7da2c0838fce40c5b5000c4d71c72708 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 64688d2615e44408ab868cb531540a84 (127.25.109.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } }
I20240502 02:02:00.856230 31137 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bec2e393d5e841c7857d6fd092aac53e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 02:02:00.857457 31143 catalog_manager.cc:5484] T bec2e393d5e841c7857d6fd092aac53e P 0db0855dad46416094c68fc878712cbb reported cstate change: config changed from index 5 to 6, VOTER 64688d2615e44408ab868cb531540a84 (127.25.109.129) evicted. New cstate: current_term: 1 leader_uuid: "0db0855dad46416094c68fc878712cbb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 02:02:00.862116 31270 tablet_service.cc:1514] Processing DeleteTablet for tablet bec2e393d5e841c7857d6fd092aac53e with delete_type TABLET_DATA_TOMBSTONED (TS 64688d2615e44408ab868cb531540a84 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48614
I20240502 02:02:00.863090   427 tablet_replica.cc:330] T bec2e393d5e841c7857d6fd092aac53e P 64688d2615e44408ab868cb531540a84: stopping tablet replica
I20240502 02:02:00.863508   427 raft_consensus.cc:2229] T bec2e393d5e841c7857d6fd092aac53e P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 02:02:00.864012   427 raft_consensus.cc:2258] T bec2e393d5e841c7857d6fd092aac53e P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 02:02:00.865953   427 ts_tablet_manager.cc:1875] T bec2e393d5e841c7857d6fd092aac53e P 64688d2615e44408ab868cb531540a84: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 02:02:00.871316   427 ts_tablet_manager.cc:1888] T bec2e393d5e841c7857d6fd092aac53e P 64688d2615e44408ab868cb531540a84: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 02:02:00.871477   427 log.cc:1196] T bec2e393d5e841c7857d6fd092aac53e P 64688d2615e44408ab868cb531540a84: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal/wals/bec2e393d5e841c7857d6fd092aac53e
I20240502 02:02:00.872406 31136 catalog_manager.cc:4830] TS 64688d2615e44408ab868cb531540a84 (127.25.109.129:33021): tablet bec2e393d5e841c7857d6fd092aac53e (table rebalance_test_table_0 [id=c8cf2d1749644bbb8b2330bcbf7cb196]) successfully deleted
I20240502 02:02:00.959704 32730 raft_consensus.cc:1050] T 1b97531b47d44bf8a7f905ef339f857a P 64688d2615e44408ab868cb531540a84: attempting to promote NON_VOTER 29755ed6e9444b2daf5807bd23451d29 to VOTER
I20240502 02:02:00.960681 32730 consensus_queue.cc:237] T 1b97531b47d44bf8a7f905ef339f857a P 64688d2615e44408ab868cb531540a84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } }
I20240502 02:02:00.963335 31558 raft_consensus.cc:1261] T 1b97531b47d44bf8a7f905ef339f857a P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Refusing update from remote peer 64688d2615e44408ab868cb531540a84: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:02:00.963649 32230 raft_consensus.cc:1261] T 1b97531b47d44bf8a7f905ef339f857a P 29755ed6e9444b2daf5807bd23451d29 [term 1 LEARNER]: Refusing update from remote peer 64688d2615e44408ab868cb531540a84: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:02:00.963958 32557 consensus_queue.cc:1035] T 1b97531b47d44bf8a7f905ef339f857a P 64688d2615e44408ab868cb531540a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:00.964066 31428 raft_consensus.cc:1261] T 1b97531b47d44bf8a7f905ef339f857a P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Refusing update from remote peer 64688d2615e44408ab868cb531540a84: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:02:00.964406 32730 consensus_queue.cc:1035] T 1b97531b47d44bf8a7f905ef339f857a P 64688d2615e44408ab868cb531540a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:00.964812   429 consensus_queue.cc:1035] T 1b97531b47d44bf8a7f905ef339f857a P 64688d2615e44408ab868cb531540a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:00.967957   429 raft_consensus.cc:2932] T 1b97531b47d44bf8a7f905ef339f857a P 64688d2615e44408ab868cb531540a84 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29755ed6e9444b2daf5807bd23451d29 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } }
I20240502 02:02:00.968752 31428 raft_consensus.cc:2932] T 1b97531b47d44bf8a7f905ef339f857a P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29755ed6e9444b2daf5807bd23451d29 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } }
I20240502 02:02:00.969828 32230 raft_consensus.cc:2932] T 1b97531b47d44bf8a7f905ef339f857a P 29755ed6e9444b2daf5807bd23451d29 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29755ed6e9444b2daf5807bd23451d29 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } }
I20240502 02:02:00.972121 31558 raft_consensus.cc:2932] T 1b97531b47d44bf8a7f905ef339f857a P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29755ed6e9444b2daf5807bd23451d29 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } }
I20240502 02:02:00.975420 31141 catalog_manager.cc:5484] T 1b97531b47d44bf8a7f905ef339f857a P 64688d2615e44408ab868cb531540a84 reported cstate change: config changed from index 4 to 5, 29755ed6e9444b2daf5807bd23451d29 (127.25.109.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "64688d2615e44408ab868cb531540a84" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 02:02:00.979346 32557 raft_consensus.cc:1050] T a10fd1feee3a4838a8fba1599a1b890d P 64688d2615e44408ab868cb531540a84: attempting to promote NON_VOTER 29755ed6e9444b2daf5807bd23451d29 to VOTER
I20240502 02:02:00.980356 32557 consensus_queue.cc:237] T a10fd1feee3a4838a8fba1599a1b890d P 64688d2615e44408ab868cb531540a84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } }
I20240502 02:02:00.981587 31292 consensus_queue.cc:237] T 1b97531b47d44bf8a7f905ef339f857a P 64688d2615e44408ab868cb531540a84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } }
I20240502 02:02:00.982964 32230 raft_consensus.cc:1261] T a10fd1feee3a4838a8fba1599a1b890d P 29755ed6e9444b2daf5807bd23451d29 [term 1 LEARNER]: Refusing update from remote peer 64688d2615e44408ab868cb531540a84: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:02:00.983290 31428 raft_consensus.cc:1261] T a10fd1feee3a4838a8fba1599a1b890d P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Refusing update from remote peer 64688d2615e44408ab868cb531540a84: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:02:00.984071 31559 raft_consensus.cc:1261] T 1b97531b47d44bf8a7f905ef339f857a P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Refusing update from remote peer 64688d2615e44408ab868cb531540a84: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:02:00.984540 31558 raft_consensus.cc:1261] T a10fd1feee3a4838a8fba1599a1b890d P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Refusing update from remote peer 64688d2615e44408ab868cb531540a84: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:02:00.984555 32230 raft_consensus.cc:1261] T 1b97531b47d44bf8a7f905ef339f857a P 29755ed6e9444b2daf5807bd23451d29 [term 1 FOLLOWER]: Refusing update from remote peer 64688d2615e44408ab868cb531540a84: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:02:00.984655   335 consensus_queue.cc:1035] T a10fd1feee3a4838a8fba1599a1b890d P 64688d2615e44408ab868cb531540a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:00.985078 32730 consensus_queue.cc:1035] T 1b97531b47d44bf8a7f905ef339f857a P 64688d2615e44408ab868cb531540a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:02:00.985517   335 consensus_queue.cc:1035] T 1b97531b47d44bf8a7f905ef339f857a P 64688d2615e44408ab868cb531540a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:02:00.985642 32730 consensus_queue.cc:1035] T a10fd1feee3a4838a8fba1599a1b890d P 64688d2615e44408ab868cb531540a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:00.986131   429 consensus_queue.cc:1035] T a10fd1feee3a4838a8fba1599a1b890d P 64688d2615e44408ab868cb531540a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:00.987957 32730 raft_consensus.cc:2932] T 1b97531b47d44bf8a7f905ef339f857a P 64688d2615e44408ab868cb531540a84 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0db0855dad46416094c68fc878712cbb (127.25.109.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } }
I20240502 02:02:00.992034 31136 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1b97531b47d44bf8a7f905ef339f857a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 02:02:00.992849 31558 raft_consensus.cc:2932] T 1b97531b47d44bf8a7f905ef339f857a P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0db0855dad46416094c68fc878712cbb (127.25.109.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } }
I20240502 02:02:00.993602 32230 raft_consensus.cc:2932] T 1b97531b47d44bf8a7f905ef339f857a P 29755ed6e9444b2daf5807bd23451d29 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0db0855dad46416094c68fc878712cbb (127.25.109.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } }
I20240502 02:02:00.994359 31150 catalog_manager.cc:5484] T 1b97531b47d44bf8a7f905ef339f857a P 64688d2615e44408ab868cb531540a84 reported cstate change: config changed from index 5 to 6, VOTER 0db0855dad46416094c68fc878712cbb (127.25.109.130) evicted. New cstate: current_term: 1 leader_uuid: "64688d2615e44408ab868cb531540a84" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 02:02:01.001873 31407 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b97531b47d44bf8a7f905ef339f857a with delete_type TABLET_DATA_TOMBSTONED (TS 0db0855dad46416094c68fc878712cbb not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:43004
I20240502 02:02:01.003542   437 tablet_replica.cc:330] T 1b97531b47d44bf8a7f905ef339f857a P 0db0855dad46416094c68fc878712cbb: stopping tablet replica
I20240502 02:02:01.004470   437 raft_consensus.cc:2229] T 1b97531b47d44bf8a7f905ef339f857a P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 02:02:01.004317   335 raft_consensus.cc:2932] T a10fd1feee3a4838a8fba1599a1b890d P 64688d2615e44408ab868cb531540a84 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29755ed6e9444b2daf5807bd23451d29 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } }
I20240502 02:02:01.005149   437 raft_consensus.cc:2258] T 1b97531b47d44bf8a7f905ef339f857a P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 02:02:01.007501   437 ts_tablet_manager.cc:1875] T 1b97531b47d44bf8a7f905ef339f857a P 0db0855dad46416094c68fc878712cbb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 02:02:01.009256 31559 raft_consensus.cc:2932] T a10fd1feee3a4838a8fba1599a1b890d P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29755ed6e9444b2daf5807bd23451d29 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } }
I20240502 02:02:01.012313 31428 raft_consensus.cc:2932] T a10fd1feee3a4838a8fba1599a1b890d P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29755ed6e9444b2daf5807bd23451d29 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } }
I20240502 02:02:01.014081 32229 raft_consensus.cc:2932] T a10fd1feee3a4838a8fba1599a1b890d P 29755ed6e9444b2daf5807bd23451d29 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29755ed6e9444b2daf5807bd23451d29 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } }
I20240502 02:02:01.016491   437 ts_tablet_manager.cc:1888] T 1b97531b47d44bf8a7f905ef339f857a P 0db0855dad46416094c68fc878712cbb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 02:02:01.016777   437 log.cc:1196] T 1b97531b47d44bf8a7f905ef339f857a P 0db0855dad46416094c68fc878712cbb: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal/wals/1b97531b47d44bf8a7f905ef339f857a
I20240502 02:02:01.018353 31137 catalog_manager.cc:4830] TS 0db0855dad46416094c68fc878712cbb (127.25.109.130:38025): tablet 1b97531b47d44bf8a7f905ef339f857a (table rebalance_test_table_2 [id=a1611faa4bb54872a21542e38fdc1e77]) successfully deleted
I20240502 02:02:01.019241 31150 catalog_manager.cc:5484] T a10fd1feee3a4838a8fba1599a1b890d P e325c0c7447f4b329994aafdee580666 reported cstate change: config changed from index 4 to 5, 29755ed6e9444b2daf5807bd23451d29 (127.25.109.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "64688d2615e44408ab868cb531540a84" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } }
I20240502 02:02:01.030164 31292 consensus_queue.cc:237] T a10fd1feee3a4838a8fba1599a1b890d P 64688d2615e44408ab868cb531540a84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } }
I20240502 02:02:01.032752 32229 raft_consensus.cc:1261] T a10fd1feee3a4838a8fba1599a1b890d P 29755ed6e9444b2daf5807bd23451d29 [term 1 FOLLOWER]: Refusing update from remote peer 64688d2615e44408ab868cb531540a84: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:02:01.033361 31428 raft_consensus.cc:1261] T a10fd1feee3a4838a8fba1599a1b890d P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Refusing update from remote peer 64688d2615e44408ab868cb531540a84: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:02:01.033448   335 consensus_queue.cc:1035] T a10fd1feee3a4838a8fba1599a1b890d P 64688d2615e44408ab868cb531540a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:02:01.034363   335 consensus_queue.cc:1035] T a10fd1feee3a4838a8fba1599a1b890d P 64688d2615e44408ab868cb531540a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:02:01.036710   429 raft_consensus.cc:2932] T a10fd1feee3a4838a8fba1599a1b890d P 64688d2615e44408ab868cb531540a84 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e325c0c7447f4b329994aafdee580666 (127.25.109.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } }
I20240502 02:02:01.037740 32229 raft_consensus.cc:2932] T a10fd1feee3a4838a8fba1599a1b890d P 29755ed6e9444b2daf5807bd23451d29 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e325c0c7447f4b329994aafdee580666 (127.25.109.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } }
I20240502 02:02:01.040174 31428 raft_consensus.cc:2932] T a10fd1feee3a4838a8fba1599a1b890d P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e325c0c7447f4b329994aafdee580666 (127.25.109.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } }
I20240502 02:02:01.040971 31136 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a10fd1feee3a4838a8fba1599a1b890d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 02:02:01.042795 31141 catalog_manager.cc:5484] T a10fd1feee3a4838a8fba1599a1b890d P 64688d2615e44408ab868cb531540a84 reported cstate change: config changed from index 5 to 6, VOTER e325c0c7447f4b329994aafdee580666 (127.25.109.131) evicted. New cstate: current_term: 1 leader_uuid: "64688d2615e44408ab868cb531540a84" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 02:02:01.047435 31538 tablet_service.cc:1514] Processing DeleteTablet for tablet a10fd1feee3a4838a8fba1599a1b890d with delete_type TABLET_DATA_TOMBSTONED (TS e325c0c7447f4b329994aafdee580666 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59068
I20240502 02:02:01.049630   438 raft_consensus.cc:1050] T 171f811ca5584cd3ab08fc25a9b7f7fc P 64688d2615e44408ab868cb531540a84: attempting to promote NON_VOTER 7da2c0838fce40c5b5000c4d71c72708 to VOTER
I20240502 02:02:01.050475   438 consensus_queue.cc:237] T 171f811ca5584cd3ab08fc25a9b7f7fc P 64688d2615e44408ab868cb531540a84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } }
I20240502 02:02:01.053294 32364 raft_consensus.cc:1261] T 171f811ca5584cd3ab08fc25a9b7f7fc P 7da2c0838fce40c5b5000c4d71c72708 [term 1 LEARNER]: Refusing update from remote peer 64688d2615e44408ab868cb531540a84: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:02:01.053309 31559 raft_consensus.cc:1261] T 171f811ca5584cd3ab08fc25a9b7f7fc P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Refusing update from remote peer 64688d2615e44408ab868cb531540a84: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:02:01.053575 31428 raft_consensus.cc:1261] T 171f811ca5584cd3ab08fc25a9b7f7fc P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Refusing update from remote peer 64688d2615e44408ab868cb531540a84: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:02:01.053968   438 consensus_queue.cc:1035] T 171f811ca5584cd3ab08fc25a9b7f7fc P 64688d2615e44408ab868cb531540a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:01.054345 32557 consensus_queue.cc:1035] T 171f811ca5584cd3ab08fc25a9b7f7fc P 64688d2615e44408ab868cb531540a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:01.054587   335 consensus_queue.cc:1035] T 171f811ca5584cd3ab08fc25a9b7f7fc P 64688d2615e44408ab868cb531540a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:01.060086   335 raft_consensus.cc:2932] T 171f811ca5584cd3ab08fc25a9b7f7fc P 64688d2615e44408ab868cb531540a84 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7da2c0838fce40c5b5000c4d71c72708 (127.25.109.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } }
I20240502 02:02:01.064131   444 tablet_replica.cc:330] T a10fd1feee3a4838a8fba1599a1b890d P e325c0c7447f4b329994aafdee580666: stopping tablet replica
I20240502 02:02:01.064695   444 raft_consensus.cc:2229] T a10fd1feee3a4838a8fba1599a1b890d P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 02:02:01.065119   444 raft_consensus.cc:2258] T a10fd1feee3a4838a8fba1599a1b890d P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 02:02:01.065811 31559 raft_consensus.cc:2932] T 171f811ca5584cd3ab08fc25a9b7f7fc P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7da2c0838fce40c5b5000c4d71c72708 (127.25.109.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } }
I20240502 02:02:01.066277 31144 catalog_manager.cc:5484] T 171f811ca5584cd3ab08fc25a9b7f7fc P 64688d2615e44408ab868cb531540a84 reported cstate change: config changed from index 4 to 5, 7da2c0838fce40c5b5000c4d71c72708 (127.25.109.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "64688d2615e44408ab868cb531540a84" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 02:02:01.066301 31428 raft_consensus.cc:2932] T 171f811ca5584cd3ab08fc25a9b7f7fc P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7da2c0838fce40c5b5000c4d71c72708 (127.25.109.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } }
I20240502 02:02:01.075047   444 ts_tablet_manager.cc:1875] T a10fd1feee3a4838a8fba1599a1b890d P e325c0c7447f4b329994aafdee580666: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 02:02:01.077704 32364 raft_consensus.cc:2932] T 171f811ca5584cd3ab08fc25a9b7f7fc P 7da2c0838fce40c5b5000c4d71c72708 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7da2c0838fce40c5b5000c4d71c72708 (127.25.109.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } }
I20240502 02:02:01.080745 31292 consensus_queue.cc:237] T 171f811ca5584cd3ab08fc25a9b7f7fc P 64688d2615e44408ab868cb531540a84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } }
I20240502 02:02:01.082254   444 ts_tablet_manager.cc:1888] T a10fd1feee3a4838a8fba1599a1b890d P e325c0c7447f4b329994aafdee580666: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 02:02:01.082448   444 log.cc:1196] T a10fd1feee3a4838a8fba1599a1b890d P e325c0c7447f4b329994aafdee580666: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/wals/a10fd1feee3a4838a8fba1599a1b890d
I20240502 02:02:01.083493 31136 catalog_manager.cc:4830] TS e325c0c7447f4b329994aafdee580666 (127.25.109.131:43937): tablet a10fd1feee3a4838a8fba1599a1b890d (table rebalance_test_table_1 [id=1e4e557617e14b7895769013935449f4]) successfully deleted
I20240502 02:02:01.084408 31559 raft_consensus.cc:1261] T 171f811ca5584cd3ab08fc25a9b7f7fc P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Refusing update from remote peer 64688d2615e44408ab868cb531540a84: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:02:01.084460 32363 raft_consensus.cc:1261] T 171f811ca5584cd3ab08fc25a9b7f7fc P 7da2c0838fce40c5b5000c4d71c72708 [term 1 FOLLOWER]: Refusing update from remote peer 64688d2615e44408ab868cb531540a84: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:02:01.085052   335 consensus_queue.cc:1035] T 171f811ca5584cd3ab08fc25a9b7f7fc P 64688d2615e44408ab868cb531540a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:02:01.087826   438 raft_consensus.cc:2932] T 171f811ca5584cd3ab08fc25a9b7f7fc P 64688d2615e44408ab868cb531540a84 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0db0855dad46416094c68fc878712cbb (127.25.109.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } }
I20240502 02:02:01.088697 32363 raft_consensus.cc:2932] T 171f811ca5584cd3ab08fc25a9b7f7fc P 7da2c0838fce40c5b5000c4d71c72708 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0db0855dad46416094c68fc878712cbb (127.25.109.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } }
I20240502 02:02:01.092624 31136 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 171f811ca5584cd3ab08fc25a9b7f7fc with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 02:02:01.093263   335 consensus_queue.cc:1035] T 171f811ca5584cd3ab08fc25a9b7f7fc P 64688d2615e44408ab868cb531540a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:02:01.094638 31146 catalog_manager.cc:5484] T 171f811ca5584cd3ab08fc25a9b7f7fc P 7da2c0838fce40c5b5000c4d71c72708 reported cstate change: config changed from index 5 to 6, VOTER 0db0855dad46416094c68fc878712cbb (127.25.109.130) evicted. New cstate: current_term: 1 leader_uuid: "64688d2615e44408ab868cb531540a84" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } }
I20240502 02:02:01.097218   454 raft_consensus.cc:1050] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666: attempting to promote NON_VOTER 7da2c0838fce40c5b5000c4d71c72708 to VOTER
I20240502 02:02:01.099262 31407 tablet_service.cc:1514] Processing DeleteTablet for tablet 171f811ca5584cd3ab08fc25a9b7f7fc with delete_type TABLET_DATA_TOMBSTONED (TS 0db0855dad46416094c68fc878712cbb not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:43004
I20240502 02:02:01.099696   437 tablet_replica.cc:330] T 171f811ca5584cd3ab08fc25a9b7f7fc P 0db0855dad46416094c68fc878712cbb: stopping tablet replica
I20240502 02:02:01.099969   437 raft_consensus.cc:2229] T 171f811ca5584cd3ab08fc25a9b7f7fc P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 02:02:01.100325   437 raft_consensus.cc:2258] T 171f811ca5584cd3ab08fc25a9b7f7fc P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 02:02:01.101441   437 ts_tablet_manager.cc:1875] T 171f811ca5584cd3ab08fc25a9b7f7fc P 0db0855dad46416094c68fc878712cbb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 02:02:01.096933 31559 raft_consensus.cc:2932] T 171f811ca5584cd3ab08fc25a9b7f7fc P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0db0855dad46416094c68fc878712cbb (127.25.109.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } }
I20240502 02:02:01.107811   454 consensus_queue.cc:237] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } }
I20240502 02:02:01.112275 32363 raft_consensus.cc:1261] T 3475ca93c949435699b590bd97e3735c P 7da2c0838fce40c5b5000c4d71c72708 [term 1 LEARNER]: Refusing update from remote peer e325c0c7447f4b329994aafdee580666: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:02:01.115216   460 consensus_queue.cc:1035] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:01.116005 31292 raft_consensus.cc:1261] T 3475ca93c949435699b590bd97e3735c P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Refusing update from remote peer e325c0c7447f4b329994aafdee580666: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:02:01.116782   337 consensus_queue.cc:1035] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:01.116981 31428 raft_consensus.cc:1261] T 3475ca93c949435699b590bd97e3735c P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Refusing update from remote peer e325c0c7447f4b329994aafdee580666: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:02:01.118630   461 consensus_queue.cc:1035] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:01.120273   337 raft_consensus.cc:2932] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7da2c0838fce40c5b5000c4d71c72708 (127.25.109.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } }
I20240502 02:02:01.120961 31292 raft_consensus.cc:2932] T 3475ca93c949435699b590bd97e3735c P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7da2c0838fce40c5b5000c4d71c72708 (127.25.109.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } }
I20240502 02:02:01.126219 32363 raft_consensus.cc:2932] T 3475ca93c949435699b590bd97e3735c P 7da2c0838fce40c5b5000c4d71c72708 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7da2c0838fce40c5b5000c4d71c72708 (127.25.109.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } }
I20240502 02:02:01.126917 31141 catalog_manager.cc:5484] T 3475ca93c949435699b590bd97e3735c P 64688d2615e44408ab868cb531540a84 reported cstate change: config changed from index 4 to 5, 7da2c0838fce40c5b5000c4d71c72708 (127.25.109.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e325c0c7447f4b329994aafdee580666" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } }
I20240502 02:02:01.127285   437 ts_tablet_manager.cc:1888] T 171f811ca5584cd3ab08fc25a9b7f7fc P 0db0855dad46416094c68fc878712cbb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 02:02:01.127458   437 log.cc:1196] T 171f811ca5584cd3ab08fc25a9b7f7fc P 0db0855dad46416094c68fc878712cbb: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal/wals/171f811ca5584cd3ab08fc25a9b7f7fc
I20240502 02:02:01.133199   419 raft_consensus.cc:1050] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb: attempting to promote NON_VOTER 29755ed6e9444b2daf5807bd23451d29 to VOTER
I20240502 02:02:01.134079   419 consensus_queue.cc:237] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } }
I20240502 02:02:01.136358 31137 catalog_manager.cc:4830] TS 0db0855dad46416094c68fc878712cbb (127.25.109.130:38025): tablet 171f811ca5584cd3ab08fc25a9b7f7fc (table rebalance_test_table_1 [id=1e4e557617e14b7895769013935449f4]) successfully deleted
I20240502 02:02:01.139400 31428 raft_consensus.cc:2932] T 3475ca93c949435699b590bd97e3735c P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7da2c0838fce40c5b5000c4d71c72708 (127.25.109.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } }
I20240502 02:02:01.142315 31559 consensus_queue.cc:237] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } }
I20240502 02:02:01.144178 32363 raft_consensus.cc:1261] T 3475ca93c949435699b590bd97e3735c P 7da2c0838fce40c5b5000c4d71c72708 [term 1 FOLLOWER]: Refusing update from remote peer e325c0c7447f4b329994aafdee580666: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:02:01.144795   454 consensus_queue.cc:1035] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:02:01.146893 31426 raft_consensus.cc:1261] T 3475ca93c949435699b590bd97e3735c P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Refusing update from remote peer e325c0c7447f4b329994aafdee580666: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:02:01.148017   337 consensus_queue.cc:1035] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:02:01.147992   460 raft_consensus.cc:2932] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 64688d2615e44408ab868cb531540a84 (127.25.109.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } }
I20240502 02:02:01.149632 32363 raft_consensus.cc:2932] T 3475ca93c949435699b590bd97e3735c P 7da2c0838fce40c5b5000c4d71c72708 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 64688d2615e44408ab868cb531540a84 (127.25.109.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } }
I20240502 02:02:01.149881 31426 raft_consensus.cc:2932] T 3475ca93c949435699b590bd97e3735c P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 64688d2615e44408ab868cb531540a84 (127.25.109.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } }
I20240502 02:02:01.151906 32229 raft_consensus.cc:1261] T 28de9be2be73409387480db617b0031c P 29755ed6e9444b2daf5807bd23451d29 [term 1 LEARNER]: Refusing update from remote peer 0db0855dad46416094c68fc878712cbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:02:01.152232 31136 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3475ca93c949435699b590bd97e3735c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 02:02:01.152593 32572 consensus_queue.cc:1035] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:01.154907 31559 raft_consensus.cc:1261] T 28de9be2be73409387480db617b0031c P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Refusing update from remote peer 0db0855dad46416094c68fc878712cbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:02:01.156451 31146 catalog_manager.cc:5484] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 reported cstate change: config changed from index 5 to 6, VOTER 64688d2615e44408ab868cb531540a84 (127.25.109.129) evicted. New cstate: current_term: 1 leader_uuid: "e325c0c7447f4b329994aafdee580666" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 02:02:01.158386 32572 consensus_queue.cc:1035] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:01.161962 31270 tablet_service.cc:1514] Processing DeleteTablet for tablet 3475ca93c949435699b590bd97e3735c with delete_type TABLET_DATA_TOMBSTONED (TS 64688d2615e44408ab868cb531540a84 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48614
I20240502 02:02:01.162462   427 tablet_replica.cc:330] T 3475ca93c949435699b590bd97e3735c P 64688d2615e44408ab868cb531540a84: stopping tablet replica
I20240502 02:02:01.162719   438 raft_consensus.cc:1050] T 9676b44f15e84ae28a64caa71f5722c6 P 64688d2615e44408ab868cb531540a84: attempting to promote NON_VOTER 7da2c0838fce40c5b5000c4d71c72708 to VOTER
I20240502 02:02:01.162722   427 raft_consensus.cc:2229] T 3475ca93c949435699b590bd97e3735c P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 02:02:01.163166   427 raft_consensus.cc:2258] T 3475ca93c949435699b590bd97e3735c P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 02:02:01.163627   438 consensus_queue.cc:237] T 9676b44f15e84ae28a64caa71f5722c6 P 64688d2615e44408ab868cb531540a84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } }
I20240502 02:02:01.164872   427 ts_tablet_manager.cc:1875] T 3475ca93c949435699b590bd97e3735c P 64688d2615e44408ab868cb531540a84: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 02:02:01.165333 31292 raft_consensus.cc:1261] T 28de9be2be73409387480db617b0031c P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Refusing update from remote peer 0db0855dad46416094c68fc878712cbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:02:01.166343   419 consensus_queue.cc:1035] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:01.168956 32572 raft_consensus.cc:2932] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29755ed6e9444b2daf5807bd23451d29 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } }
I20240502 02:02:01.171906 31559 raft_consensus.cc:1261] T 9676b44f15e84ae28a64caa71f5722c6 P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Refusing update from remote peer 64688d2615e44408ab868cb531540a84: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:02:01.172823 32363 raft_consensus.cc:1261] T 9676b44f15e84ae28a64caa71f5722c6 P 7da2c0838fce40c5b5000c4d71c72708 [term 1 LEARNER]: Refusing update from remote peer 64688d2615e44408ab868cb531540a84: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:02:01.172658 32229 raft_consensus.cc:2932] T 28de9be2be73409387480db617b0031c P 29755ed6e9444b2daf5807bd23451d29 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29755ed6e9444b2daf5807bd23451d29 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } }
I20240502 02:02:01.173564   438 consensus_queue.cc:1035] T 9676b44f15e84ae28a64caa71f5722c6 P 64688d2615e44408ab868cb531540a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:01.173592 31141 catalog_manager.cc:5484] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb reported cstate change: config changed from index 4 to 5, 29755ed6e9444b2daf5807bd23451d29 (127.25.109.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0db0855dad46416094c68fc878712cbb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 02:02:01.176299 31428 raft_consensus.cc:1261] T 9676b44f15e84ae28a64caa71f5722c6 P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Refusing update from remote peer 64688d2615e44408ab868cb531540a84: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:02:01.176903   438 consensus_queue.cc:1035] T 9676b44f15e84ae28a64caa71f5722c6 P 64688d2615e44408ab868cb531540a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:01.177254   335 consensus_queue.cc:1035] T 9676b44f15e84ae28a64caa71f5722c6 P 64688d2615e44408ab868cb531540a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:01.178948 31558 raft_consensus.cc:2932] T 28de9be2be73409387480db617b0031c P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29755ed6e9444b2daf5807bd23451d29 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } }
I20240502 02:02:01.180122 31292 raft_consensus.cc:2932] T 28de9be2be73409387480db617b0031c P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29755ed6e9444b2daf5807bd23451d29 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } }
I20240502 02:02:01.184911   427 ts_tablet_manager.cc:1888] T 3475ca93c949435699b590bd97e3735c P 64688d2615e44408ab868cb531540a84: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 02:02:01.185127   427 log.cc:1196] T 3475ca93c949435699b590bd97e3735c P 64688d2615e44408ab868cb531540a84: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal/wals/3475ca93c949435699b590bd97e3735c
I20240502 02:02:01.187763 31136 catalog_manager.cc:4830] TS 64688d2615e44408ab868cb531540a84 (127.25.109.129:33021): tablet 3475ca93c949435699b590bd97e3735c (table rebalance_test_table_0 [id=c8cf2d1749644bbb8b2330bcbf7cb196]) successfully deleted
I20240502 02:02:01.192654   335 raft_consensus.cc:2932] T 9676b44f15e84ae28a64caa71f5722c6 P 64688d2615e44408ab868cb531540a84 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7da2c0838fce40c5b5000c4d71c72708 (127.25.109.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } }
I20240502 02:02:01.193416 31559 raft_consensus.cc:2932] T 9676b44f15e84ae28a64caa71f5722c6 P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7da2c0838fce40c5b5000c4d71c72708 (127.25.109.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } }
I20240502 02:02:01.197551 32363 raft_consensus.cc:2932] T 9676b44f15e84ae28a64caa71f5722c6 P 7da2c0838fce40c5b5000c4d71c72708 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7da2c0838fce40c5b5000c4d71c72708 (127.25.109.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } }
I20240502 02:02:01.198968 31150 catalog_manager.cc:5484] T 9676b44f15e84ae28a64caa71f5722c6 P e325c0c7447f4b329994aafdee580666 reported cstate change: config changed from index 4 to 5, 7da2c0838fce40c5b5000c4d71c72708 (127.25.109.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "64688d2615e44408ab868cb531540a84" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } }
I20240502 02:02:01.200644 31428 consensus_queue.cc:237] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } }
I20240502 02:02:01.203548 31426 raft_consensus.cc:2932] T 9676b44f15e84ae28a64caa71f5722c6 P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7da2c0838fce40c5b5000c4d71c72708 (127.25.109.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } }
I20240502 02:02:01.207270 31559 raft_consensus.cc:1261] T 28de9be2be73409387480db617b0031c P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Refusing update from remote peer 0db0855dad46416094c68fc878712cbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:02:01.208544 32229 raft_consensus.cc:1261] T 28de9be2be73409387480db617b0031c P 29755ed6e9444b2daf5807bd23451d29 [term 1 FOLLOWER]: Refusing update from remote peer 0db0855dad46416094c68fc878712cbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:02:01.209385 31292 consensus_queue.cc:237] T 9676b44f15e84ae28a64caa71f5722c6 P 64688d2615e44408ab868cb531540a84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } }
I20240502 02:02:01.211373   419 consensus_queue.cc:1035] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:02:01.211829   324 consensus_queue.cc:1035] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:02:01.212960 32363 raft_consensus.cc:1261] T 9676b44f15e84ae28a64caa71f5722c6 P 7da2c0838fce40c5b5000c4d71c72708 [term 1 FOLLOWER]: Refusing update from remote peer 64688d2615e44408ab868cb531540a84: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:02:01.213063 31428 raft_consensus.cc:1261] T 9676b44f15e84ae28a64caa71f5722c6 P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Refusing update from remote peer 64688d2615e44408ab868cb531540a84: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:02:01.213680   335 consensus_queue.cc:1035] T 9676b44f15e84ae28a64caa71f5722c6 P 64688d2615e44408ab868cb531540a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:02:01.214049 32730 consensus_queue.cc:1035] T 9676b44f15e84ae28a64caa71f5722c6 P 64688d2615e44408ab868cb531540a84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:02:01.214776   419 raft_consensus.cc:2932] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 64688d2615e44408ab868cb531540a84 (127.25.109.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } }
I20240502 02:02:01.215462 31559 raft_consensus.cc:2932] T 28de9be2be73409387480db617b0031c P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 64688d2615e44408ab868cb531540a84 (127.25.109.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } }
I20240502 02:02:01.217200 32730 raft_consensus.cc:2932] T 9676b44f15e84ae28a64caa71f5722c6 P 64688d2615e44408ab868cb531540a84 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e325c0c7447f4b329994aafdee580666 (127.25.109.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } }
I20240502 02:02:01.219735 31137 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 28de9be2be73409387480db617b0031c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 02:02:01.220947 32363 raft_consensus.cc:2932] T 9676b44f15e84ae28a64caa71f5722c6 P 7da2c0838fce40c5b5000c4d71c72708 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e325c0c7447f4b329994aafdee580666 (127.25.109.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } }
I20240502 02:02:01.220309 32229 raft_consensus.cc:2932] T 28de9be2be73409387480db617b0031c P 29755ed6e9444b2daf5807bd23451d29 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 64688d2615e44408ab868cb531540a84 (127.25.109.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } }
I20240502 02:02:01.222280 31150 catalog_manager.cc:5484] T 28de9be2be73409387480db617b0031c P e325c0c7447f4b329994aafdee580666 reported cstate change: config changed from index 5 to 6, VOTER 64688d2615e44408ab868cb531540a84 (127.25.109.129) evicted. New cstate: current_term: 1 leader_uuid: "0db0855dad46416094c68fc878712cbb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } }
I20240502 02:02:01.221529 31428 raft_consensus.cc:2932] T 9676b44f15e84ae28a64caa71f5722c6 P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e325c0c7447f4b329994aafdee580666 (127.25.109.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } }
I20240502 02:02:01.226595 31150 catalog_manager.cc:5484] T 9676b44f15e84ae28a64caa71f5722c6 P 7da2c0838fce40c5b5000c4d71c72708 reported cstate change: config changed from index 5 to 6, VOTER e325c0c7447f4b329994aafdee580666 (127.25.109.131) evicted. New cstate: current_term: 1 leader_uuid: "64688d2615e44408ab868cb531540a84" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } }
I20240502 02:02:01.227092 31136 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9676b44f15e84ae28a64caa71f5722c6 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 02:02:01.227383 31270 tablet_service.cc:1514] Processing DeleteTablet for tablet 28de9be2be73409387480db617b0031c with delete_type TABLET_DATA_TOMBSTONED (TS 64688d2615e44408ab868cb531540a84 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48614
I20240502 02:02:01.227788   427 tablet_replica.cc:330] T 28de9be2be73409387480db617b0031c P 64688d2615e44408ab868cb531540a84: stopping tablet replica
I20240502 02:02:01.228080   427 raft_consensus.cc:2229] T 28de9be2be73409387480db617b0031c P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 02:02:01.228423   427 raft_consensus.cc:2258] T 28de9be2be73409387480db617b0031c P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 02:02:01.230158   427 ts_tablet_manager.cc:1875] T 28de9be2be73409387480db617b0031c P 64688d2615e44408ab868cb531540a84: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 02:02:01.235028 31538 tablet_service.cc:1514] Processing DeleteTablet for tablet 9676b44f15e84ae28a64caa71f5722c6 with delete_type TABLET_DATA_TOMBSTONED (TS e325c0c7447f4b329994aafdee580666 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59068
I20240502 02:02:01.235558   444 tablet_replica.cc:330] T 9676b44f15e84ae28a64caa71f5722c6 P e325c0c7447f4b329994aafdee580666: stopping tablet replica
I20240502 02:02:01.235780   444 raft_consensus.cc:2229] T 9676b44f15e84ae28a64caa71f5722c6 P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 02:02:01.236236   444 raft_consensus.cc:2258] T 9676b44f15e84ae28a64caa71f5722c6 P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 02:02:01.237676   444 ts_tablet_manager.cc:1875] T 9676b44f15e84ae28a64caa71f5722c6 P e325c0c7447f4b329994aafdee580666: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 02:02:01.238840   427 ts_tablet_manager.cc:1888] T 28de9be2be73409387480db617b0031c P 64688d2615e44408ab868cb531540a84: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 02:02:01.238981   427 log.cc:1196] T 28de9be2be73409387480db617b0031c P 64688d2615e44408ab868cb531540a84: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal/wals/28de9be2be73409387480db617b0031c
I20240502 02:02:01.239828 31136 catalog_manager.cc:4830] TS 64688d2615e44408ab868cb531540a84 (127.25.109.129:33021): tablet 28de9be2be73409387480db617b0031c (table rebalance_test_table_0 [id=c8cf2d1749644bbb8b2330bcbf7cb196]) successfully deleted
I20240502 02:02:01.244508   444 ts_tablet_manager.cc:1888] T 9676b44f15e84ae28a64caa71f5722c6 P e325c0c7447f4b329994aafdee580666: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 02:02:01.244670   444 log.cc:1196] T 9676b44f15e84ae28a64caa71f5722c6 P e325c0c7447f4b329994aafdee580666: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/wals/9676b44f15e84ae28a64caa71f5722c6
I20240502 02:02:01.245484 31136 catalog_manager.cc:4830] TS e325c0c7447f4b329994aafdee580666 (127.25.109.131:43937): tablet 9676b44f15e84ae28a64caa71f5722c6 (table rebalance_test_table_2 [id=a1611faa4bb54872a21542e38fdc1e77]) successfully deleted
I20240502 02:02:01.284693   323 raft_consensus.cc:1050] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb: attempting to promote NON_VOTER 29755ed6e9444b2daf5807bd23451d29 to VOTER
I20240502 02:02:01.285602   323 consensus_queue.cc:237] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } }
I20240502 02:02:01.288573 32229 raft_consensus.cc:1261] T fecea55aa2ab478d91bcad93eca93aef P 29755ed6e9444b2daf5807bd23451d29 [term 1 LEARNER]: Refusing update from remote peer 0db0855dad46416094c68fc878712cbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:02:01.288775 31559 raft_consensus.cc:1261] T fecea55aa2ab478d91bcad93eca93aef P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Refusing update from remote peer 0db0855dad46416094c68fc878712cbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:02:01.288977 31292 raft_consensus.cc:1261] T fecea55aa2ab478d91bcad93eca93aef P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Refusing update from remote peer 0db0855dad46416094c68fc878712cbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:02:01.289285 32572 consensus_queue.cc:1035] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:01.290367 32572 consensus_queue.cc:1035] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:01.290920 32572 consensus_queue.cc:1035] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:01.301805   419 raft_consensus.cc:2932] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29755ed6e9444b2daf5807bd23451d29 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } }
I20240502 02:02:01.303385 31559 raft_consensus.cc:2932] T fecea55aa2ab478d91bcad93eca93aef P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29755ed6e9444b2daf5807bd23451d29 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } }
I20240502 02:02:01.303292 32229 raft_consensus.cc:2932] T fecea55aa2ab478d91bcad93eca93aef P 29755ed6e9444b2daf5807bd23451d29 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29755ed6e9444b2daf5807bd23451d29 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } }
I20240502 02:02:01.307126 31292 raft_consensus.cc:2932] T fecea55aa2ab478d91bcad93eca93aef P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 29755ed6e9444b2daf5807bd23451d29 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } }
I20240502 02:02:01.308247 31141 catalog_manager.cc:5484] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb reported cstate change: config changed from index 4 to 5, 29755ed6e9444b2daf5807bd23451d29 (127.25.109.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0db0855dad46416094c68fc878712cbb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 02:02:01.314148 31428 consensus_queue.cc:237] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } }
I20240502 02:02:01.316298 32229 raft_consensus.cc:1261] T fecea55aa2ab478d91bcad93eca93aef P 29755ed6e9444b2daf5807bd23451d29 [term 1 FOLLOWER]: Refusing update from remote peer 0db0855dad46416094c68fc878712cbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:02:01.317309 31292 raft_consensus.cc:1261] T fecea55aa2ab478d91bcad93eca93aef P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Refusing update from remote peer 0db0855dad46416094c68fc878712cbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:02:01.317395   419 consensus_queue.cc:1035] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:02:01.318006   419 consensus_queue.cc:1035] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:02:01.320411 32572 raft_consensus.cc:2932] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e325c0c7447f4b329994aafdee580666 (127.25.109.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } }
I20240502 02:02:01.321169 32229 raft_consensus.cc:2932] T fecea55aa2ab478d91bcad93eca93aef P 29755ed6e9444b2daf5807bd23451d29 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e325c0c7447f4b329994aafdee580666 (127.25.109.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } }
I20240502 02:02:01.323457 31292 raft_consensus.cc:2932] T fecea55aa2ab478d91bcad93eca93aef P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e325c0c7447f4b329994aafdee580666 (127.25.109.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } }
I20240502 02:02:01.324982 31137 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fecea55aa2ab478d91bcad93eca93aef with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 02:02:01.328567 31146 catalog_manager.cc:5484] T fecea55aa2ab478d91bcad93eca93aef P 29755ed6e9444b2daf5807bd23451d29 reported cstate change: config changed from index 5 to 6, VOTER e325c0c7447f4b329994aafdee580666 (127.25.109.131) evicted. New cstate: current_term: 1 leader_uuid: "0db0855dad46416094c68fc878712cbb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } }
I20240502 02:02:01.332993 31538 tablet_service.cc:1514] Processing DeleteTablet for tablet fecea55aa2ab478d91bcad93eca93aef with delete_type TABLET_DATA_TOMBSTONED (TS e325c0c7447f4b329994aafdee580666 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59068
I20240502 02:02:01.333420   444 tablet_replica.cc:330] T fecea55aa2ab478d91bcad93eca93aef P e325c0c7447f4b329994aafdee580666: stopping tablet replica
I20240502 02:02:01.333649   444 raft_consensus.cc:2229] T fecea55aa2ab478d91bcad93eca93aef P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 02:02:01.334056   444 raft_consensus.cc:2258] T fecea55aa2ab478d91bcad93eca93aef P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 02:02:01.335220   444 ts_tablet_manager.cc:1875] T fecea55aa2ab478d91bcad93eca93aef P e325c0c7447f4b329994aafdee580666: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 02:02:01.341120   444 ts_tablet_manager.cc:1888] T fecea55aa2ab478d91bcad93eca93aef P e325c0c7447f4b329994aafdee580666: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 02:02:01.341274   444 log.cc:1196] T fecea55aa2ab478d91bcad93eca93aef P e325c0c7447f4b329994aafdee580666: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/wals/fecea55aa2ab478d91bcad93eca93aef
I20240502 02:02:01.342000 31136 catalog_manager.cc:4830] TS e325c0c7447f4b329994aafdee580666 (127.25.109.131:43937): tablet fecea55aa2ab478d91bcad93eca93aef (table rebalance_test_table_1 [id=1e4e557617e14b7895769013935449f4]) successfully deleted
I20240502 02:02:01.437537 31407 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:02:01.446238 31270 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:02:01.449575 32210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:02:01.455197 32343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:02:01.460012 31538 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:01.641930 31407 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:02:01.654362 32210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:02:01.668426 31538 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:01.668876 31270 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:02:01.669463 32343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:02:01.753921 31559 consensus_queue.cc:237] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: NON_VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { promote: true } }
I20240502 02:02:01.756114 32363 raft_consensus.cc:1261] T 3475ca93c949435699b590bd97e3735c P 7da2c0838fce40c5b5000c4d71c72708 [term 1 FOLLOWER]: Refusing update from remote peer e325c0c7447f4b329994aafdee580666: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 02:02:01.756206 31426 raft_consensus.cc:1261] T 3475ca93c949435699b590bd97e3735c P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Refusing update from remote peer e325c0c7447f4b329994aafdee580666: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 02:02:01.756450 31494 consensus_peers.cc:485] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 -> Peer 64688d2615e44408ab868cb531540a84 (127.25.109.129:33021): Couldn't send request to peer 64688d2615e44408ab868cb531540a84. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:02:01.756824   460 consensus_queue.cc:1035] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 02:02:01.757143   591 consensus_queue.cc:1035] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 02:02:01.759779   590 raft_consensus.cc:2932] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 64688d2615e44408ab868cb531540a84 (127.25.109.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: NON_VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { promote: true } } }
I20240502 02:02:01.760548 32363 raft_consensus.cc:2932] T 3475ca93c949435699b590bd97e3735c P 7da2c0838fce40c5b5000c4d71c72708 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 64688d2615e44408ab868cb531540a84 (127.25.109.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: NON_VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { promote: true } } }
I20240502 02:02:01.760543 31426 raft_consensus.cc:2932] T 3475ca93c949435699b590bd97e3735c P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 64688d2615e44408ab868cb531540a84 (127.25.109.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: NON_VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { promote: true } } }
I20240502 02:02:01.764755 31150 catalog_manager.cc:5484] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 reported cstate change: config changed from index 6 to 7, NON_VOTER 64688d2615e44408ab868cb531540a84 (127.25.109.129) added. New cstate: current_term: 1 leader_uuid: "e325c0c7447f4b329994aafdee580666" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: NON_VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 02:02:01.782655 31426 consensus_queue.cc:237] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: NON_VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { promote: true } }
I20240502 02:02:01.784940 31559 raft_consensus.cc:1261] T 28de9be2be73409387480db617b0031c P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Refusing update from remote peer 0db0855dad46416094c68fc878712cbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 02:02:01.784940 32229 raft_consensus.cc:1261] T 28de9be2be73409387480db617b0031c P 29755ed6e9444b2daf5807bd23451d29 [term 1 FOLLOWER]: Refusing update from remote peer 0db0855dad46416094c68fc878712cbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 02:02:01.785604   419 consensus_queue.cc:1035] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 02:02:01.785939   323 consensus_queue.cc:1035] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 02:02:01.786257 31361 consensus_peers.cc:485] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb -> Peer 64688d2615e44408ab868cb531540a84 (127.25.109.129:33021): Couldn't send request to peer 64688d2615e44408ab868cb531540a84. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:02:01.788395 32572 raft_consensus.cc:2932] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 64688d2615e44408ab868cb531540a84 (127.25.109.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: NON_VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { promote: true } } }
I20240502 02:02:01.789176 32229 raft_consensus.cc:2932] T 28de9be2be73409387480db617b0031c P 29755ed6e9444b2daf5807bd23451d29 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 64688d2615e44408ab868cb531540a84 (127.25.109.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: NON_VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { promote: true } } }
I20240502 02:02:01.792297 31559 raft_consensus.cc:2932] T 28de9be2be73409387480db617b0031c P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 64688d2615e44408ab868cb531540a84 (127.25.109.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: NON_VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { promote: true } } }
I20240502 02:02:01.794366 31150 catalog_manager.cc:5484] T 28de9be2be73409387480db617b0031c P 29755ed6e9444b2daf5807bd23451d29 reported cstate change: config changed from index 6 to 7, NON_VOTER 64688d2615e44408ab868cb531540a84 (127.25.109.129) added. New cstate: current_term: 1 leader_uuid: "0db0855dad46416094c68fc878712cbb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: NON_VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { promote: true } } }
I20240502 02:02:01.807433 31426 consensus_queue.cc:237] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: NON_VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { promote: true } }
I20240502 02:02:01.809378 32229 raft_consensus.cc:1261] T fecea55aa2ab478d91bcad93eca93aef P 29755ed6e9444b2daf5807bd23451d29 [term 1 FOLLOWER]: Refusing update from remote peer 0db0855dad46416094c68fc878712cbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 02:02:01.809689 31361 consensus_peers.cc:485] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb -> Peer e325c0c7447f4b329994aafdee580666 (127.25.109.131:43937): Couldn't send request to peer e325c0c7447f4b329994aafdee580666. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:02:01.809809 31291 raft_consensus.cc:1261] T fecea55aa2ab478d91bcad93eca93aef P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Refusing update from remote peer 0db0855dad46416094c68fc878712cbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 02:02:01.810104   419 consensus_queue.cc:1035] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 02:02:01.810410 32572 consensus_queue.cc:1035] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 02:02:01.812651   323 raft_consensus.cc:2932] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e325c0c7447f4b329994aafdee580666 (127.25.109.131) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: NON_VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { promote: true } } }
I20240502 02:02:01.813359 31291 raft_consensus.cc:2932] T fecea55aa2ab478d91bcad93eca93aef P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e325c0c7447f4b329994aafdee580666 (127.25.109.131) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: NON_VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { promote: true } } }
I20240502 02:02:01.816499 32229 raft_consensus.cc:2932] T fecea55aa2ab478d91bcad93eca93aef P 29755ed6e9444b2daf5807bd23451d29 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e325c0c7447f4b329994aafdee580666 (127.25.109.131) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: NON_VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { promote: true } } }
I20240502 02:02:01.818249 31146 catalog_manager.cc:5484] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb reported cstate change: config changed from index 6 to 7, NON_VOTER e325c0c7447f4b329994aafdee580666 (127.25.109.131) added. New cstate: current_term: 1 leader_uuid: "0db0855dad46416094c68fc878712cbb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: NON_VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 02:02:02.171036   639 ts_tablet_manager.cc:902] T 3475ca93c949435699b590bd97e3735c P 64688d2615e44408ab868cb531540a84: Initiating tablet copy from peer e325c0c7447f4b329994aafdee580666 (127.25.109.131:43937)
I20240502 02:02:02.172662   639 tablet_copy_client.cc:308] T 3475ca93c949435699b590bd97e3735c P 64688d2615e44408ab868cb531540a84: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.131:43937
I20240502 02:02:02.173544 31581 tablet_copy_service.cc:140] P e325c0c7447f4b329994aafdee580666: Received BeginTabletCopySession request for tablet 3475ca93c949435699b590bd97e3735c from peer 64688d2615e44408ab868cb531540a84 ({username='slave'} at 127.25.109.129:40369)
I20240502 02:02:02.173722 31581 tablet_copy_service.cc:161] P e325c0c7447f4b329994aafdee580666: Beginning new tablet copy session on tablet 3475ca93c949435699b590bd97e3735c from peer 64688d2615e44408ab868cb531540a84 at {username='slave'} at 127.25.109.129:40369: session id = 64688d2615e44408ab868cb531540a84-3475ca93c949435699b590bd97e3735c
I20240502 02:02:02.175705 31581 tablet_copy_source_session.cc:205] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 02:02:02.177170   639 ts_tablet_manager.cc:1875] T 3475ca93c949435699b590bd97e3735c P 64688d2615e44408ab868cb531540a84: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 02:02:02.181923   639 ts_tablet_manager.cc:1888] T 3475ca93c949435699b590bd97e3735c P 64688d2615e44408ab868cb531540a84: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 02:02:02.182235   639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3475ca93c949435699b590bd97e3735c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:02.184919   639 tablet_copy_client.cc:783] T 3475ca93c949435699b590bd97e3735c P 64688d2615e44408ab868cb531540a84: tablet copy: Starting download of 0 data blocks...
I20240502 02:02:02.185226   639 tablet_copy_client.cc:647] T 3475ca93c949435699b590bd97e3735c P 64688d2615e44408ab868cb531540a84: tablet copy: Starting download of 1 WAL segments...
I20240502 02:02:02.187341   639 tablet_copy_client.cc:523] T 3475ca93c949435699b590bd97e3735c P 64688d2615e44408ab868cb531540a84: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:02:02.190508   639 tablet_bootstrap.cc:492] T 3475ca93c949435699b590bd97e3735c P 64688d2615e44408ab868cb531540a84: Bootstrap starting.
I20240502 02:02:02.225526   639 tablet_bootstrap.cc:492] T 3475ca93c949435699b590bd97e3735c P 64688d2615e44408ab868cb531540a84: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:02:02.225946   639 tablet_bootstrap.cc:492] T 3475ca93c949435699b590bd97e3735c P 64688d2615e44408ab868cb531540a84: Bootstrap complete.
I20240502 02:02:02.226203   639 ts_tablet_manager.cc:1367] T 3475ca93c949435699b590bd97e3735c P 64688d2615e44408ab868cb531540a84: Time spent bootstrapping tablet: real 0.036s	user 0.032s	sys 0.004s
I20240502 02:02:02.226946   639 raft_consensus.cc:348] T 3475ca93c949435699b590bd97e3735c P 64688d2615e44408ab868cb531540a84 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: NON_VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { promote: true } }
I20240502 02:02:02.227175   639 raft_consensus.cc:729] T 3475ca93c949435699b590bd97e3735c P 64688d2615e44408ab868cb531540a84 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 64688d2615e44408ab868cb531540a84, State: Initialized, Role: LEARNER
I20240502 02:02:02.227371   639 consensus_queue.cc:260] T 3475ca93c949435699b590bd97e3735c P 64688d2615e44408ab868cb531540a84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: NON_VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { promote: true } }
I20240502 02:02:02.227984   639 ts_tablet_manager.cc:1398] T 3475ca93c949435699b590bd97e3735c P 64688d2615e44408ab868cb531540a84: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240502 02:02:02.228857 31581 tablet_copy_service.cc:342] P e325c0c7447f4b329994aafdee580666: Request end of tablet copy session 64688d2615e44408ab868cb531540a84-3475ca93c949435699b590bd97e3735c received from {username='slave'} at 127.25.109.129:40369
I20240502 02:02:02.229000 31581 tablet_copy_service.cc:433] P e325c0c7447f4b329994aafdee580666: ending tablet copy session 64688d2615e44408ab868cb531540a84-3475ca93c949435699b590bd97e3735c on tablet 3475ca93c949435699b590bd97e3735c with peer 64688d2615e44408ab868cb531540a84
I20240502 02:02:02.252497   645 ts_tablet_manager.cc:902] T fecea55aa2ab478d91bcad93eca93aef P e325c0c7447f4b329994aafdee580666: Initiating tablet copy from peer 0db0855dad46416094c68fc878712cbb (127.25.109.130:38025)
I20240502 02:02:02.254069   645 tablet_copy_client.cc:308] T fecea55aa2ab478d91bcad93eca93aef P e325c0c7447f4b329994aafdee580666: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.130:38025
I20240502 02:02:02.255066 31448 tablet_copy_service.cc:140] P 0db0855dad46416094c68fc878712cbb: Received BeginTabletCopySession request for tablet fecea55aa2ab478d91bcad93eca93aef from peer e325c0c7447f4b329994aafdee580666 ({username='slave'} at 127.25.109.131:34895)
I20240502 02:02:02.255267 31448 tablet_copy_service.cc:161] P 0db0855dad46416094c68fc878712cbb: Beginning new tablet copy session on tablet fecea55aa2ab478d91bcad93eca93aef from peer e325c0c7447f4b329994aafdee580666 at {username='slave'} at 127.25.109.131:34895: session id = e325c0c7447f4b329994aafdee580666-fecea55aa2ab478d91bcad93eca93aef
I20240502 02:02:02.257299 31448 tablet_copy_source_session.cc:205] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 02:02:02.258749   645 ts_tablet_manager.cc:1875] T fecea55aa2ab478d91bcad93eca93aef P e325c0c7447f4b329994aafdee580666: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 02:02:02.266378   645 ts_tablet_manager.cc:1888] T fecea55aa2ab478d91bcad93eca93aef P e325c0c7447f4b329994aafdee580666: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 02:02:02.266741   645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fecea55aa2ab478d91bcad93eca93aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:02.270570   645 tablet_copy_client.cc:783] T fecea55aa2ab478d91bcad93eca93aef P e325c0c7447f4b329994aafdee580666: tablet copy: Starting download of 0 data blocks...
I20240502 02:02:02.270869   645 tablet_copy_client.cc:647] T fecea55aa2ab478d91bcad93eca93aef P e325c0c7447f4b329994aafdee580666: tablet copy: Starting download of 1 WAL segments...
I20240502 02:02:02.275462   645 tablet_copy_client.cc:523] T fecea55aa2ab478d91bcad93eca93aef P e325c0c7447f4b329994aafdee580666: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:02:02.279068   645 tablet_bootstrap.cc:492] T fecea55aa2ab478d91bcad93eca93aef P e325c0c7447f4b329994aafdee580666: Bootstrap starting.
I20240502 02:02:02.279269   639 ts_tablet_manager.cc:902] T 28de9be2be73409387480db617b0031c P 64688d2615e44408ab868cb531540a84: Initiating tablet copy from peer 0db0855dad46416094c68fc878712cbb (127.25.109.130:38025)
I20240502 02:02:02.286942   639 tablet_copy_client.cc:308] T 28de9be2be73409387480db617b0031c P 64688d2615e44408ab868cb531540a84: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.130:38025
I20240502 02:02:02.288084 31448 tablet_copy_service.cc:140] P 0db0855dad46416094c68fc878712cbb: Received BeginTabletCopySession request for tablet 28de9be2be73409387480db617b0031c from peer 64688d2615e44408ab868cb531540a84 ({username='slave'} at 127.25.109.129:51003)
I20240502 02:02:02.288286 31448 tablet_copy_service.cc:161] P 0db0855dad46416094c68fc878712cbb: Beginning new tablet copy session on tablet 28de9be2be73409387480db617b0031c from peer 64688d2615e44408ab868cb531540a84 at {username='slave'} at 127.25.109.129:51003: session id = 64688d2615e44408ab868cb531540a84-28de9be2be73409387480db617b0031c
I20240502 02:02:02.291574 31448 tablet_copy_source_session.cc:205] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 02:02:02.293251   639 ts_tablet_manager.cc:1875] T 28de9be2be73409387480db617b0031c P 64688d2615e44408ab868cb531540a84: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 02:02:02.300225   639 ts_tablet_manager.cc:1888] T 28de9be2be73409387480db617b0031c P 64688d2615e44408ab868cb531540a84: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 02:02:02.300642   639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28de9be2be73409387480db617b0031c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:02.303896   639 tablet_copy_client.cc:783] T 28de9be2be73409387480db617b0031c P 64688d2615e44408ab868cb531540a84: tablet copy: Starting download of 0 data blocks...
I20240502 02:02:02.304129   639 tablet_copy_client.cc:647] T 28de9be2be73409387480db617b0031c P 64688d2615e44408ab868cb531540a84: tablet copy: Starting download of 1 WAL segments...
I20240502 02:02:02.306035   639 tablet_copy_client.cc:523] T 28de9be2be73409387480db617b0031c P 64688d2615e44408ab868cb531540a84: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:02:02.309506   639 tablet_bootstrap.cc:492] T 28de9be2be73409387480db617b0031c P 64688d2615e44408ab868cb531540a84: Bootstrap starting.
I20240502 02:02:02.326561   645 tablet_bootstrap.cc:492] T fecea55aa2ab478d91bcad93eca93aef P e325c0c7447f4b329994aafdee580666: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:02:02.327190   645 tablet_bootstrap.cc:492] T fecea55aa2ab478d91bcad93eca93aef P e325c0c7447f4b329994aafdee580666: Bootstrap complete.
I20240502 02:02:02.327530   645 ts_tablet_manager.cc:1367] T fecea55aa2ab478d91bcad93eca93aef P e325c0c7447f4b329994aafdee580666: Time spent bootstrapping tablet: real 0.049s	user 0.041s	sys 0.003s
I20240502 02:02:02.328423   645 raft_consensus.cc:348] T fecea55aa2ab478d91bcad93eca93aef P e325c0c7447f4b329994aafdee580666 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: NON_VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { promote: true } }
I20240502 02:02:02.328694   645 raft_consensus.cc:729] T fecea55aa2ab478d91bcad93eca93aef P e325c0c7447f4b329994aafdee580666 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e325c0c7447f4b329994aafdee580666, State: Initialized, Role: LEARNER
I20240502 02:02:02.328928   645 consensus_queue.cc:260] T fecea55aa2ab478d91bcad93eca93aef P e325c0c7447f4b329994aafdee580666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: NON_VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { promote: true } }
I20240502 02:02:02.329540   645 ts_tablet_manager.cc:1398] T fecea55aa2ab478d91bcad93eca93aef P e325c0c7447f4b329994aafdee580666: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240502 02:02:02.330436 31448 tablet_copy_service.cc:342] P 0db0855dad46416094c68fc878712cbb: Request end of tablet copy session e325c0c7447f4b329994aafdee580666-fecea55aa2ab478d91bcad93eca93aef received from {username='slave'} at 127.25.109.131:34895
I20240502 02:02:02.330646 31448 tablet_copy_service.cc:433] P 0db0855dad46416094c68fc878712cbb: ending tablet copy session e325c0c7447f4b329994aafdee580666-fecea55aa2ab478d91bcad93eca93aef on tablet fecea55aa2ab478d91bcad93eca93aef with peer e325c0c7447f4b329994aafdee580666
I20240502 02:02:02.348659   639 tablet_bootstrap.cc:492] T 28de9be2be73409387480db617b0031c P 64688d2615e44408ab868cb531540a84: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:02:02.349089   639 tablet_bootstrap.cc:492] T 28de9be2be73409387480db617b0031c P 64688d2615e44408ab868cb531540a84: Bootstrap complete.
I20240502 02:02:02.349339   639 ts_tablet_manager.cc:1367] T 28de9be2be73409387480db617b0031c P 64688d2615e44408ab868cb531540a84: Time spent bootstrapping tablet: real 0.040s	user 0.035s	sys 0.004s
I20240502 02:02:02.350056   639 raft_consensus.cc:348] T 28de9be2be73409387480db617b0031c P 64688d2615e44408ab868cb531540a84 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: NON_VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { promote: true } }
I20240502 02:02:02.350262   639 raft_consensus.cc:729] T 28de9be2be73409387480db617b0031c P 64688d2615e44408ab868cb531540a84 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 64688d2615e44408ab868cb531540a84, State: Initialized, Role: LEARNER
I20240502 02:02:02.350466   639 consensus_queue.cc:260] T 28de9be2be73409387480db617b0031c P 64688d2615e44408ab868cb531540a84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: NON_VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { promote: true } }
I20240502 02:02:02.350984   639 ts_tablet_manager.cc:1398] T 28de9be2be73409387480db617b0031c P 64688d2615e44408ab868cb531540a84: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 02:02:02.351799 31448 tablet_copy_service.cc:342] P 0db0855dad46416094c68fc878712cbb: Request end of tablet copy session 64688d2615e44408ab868cb531540a84-28de9be2be73409387480db617b0031c received from {username='slave'} at 127.25.109.129:51003
I20240502 02:02:02.352041 31448 tablet_copy_service.cc:433] P 0db0855dad46416094c68fc878712cbb: ending tablet copy session 64688d2615e44408ab868cb531540a84-28de9be2be73409387480db617b0031c on tablet 28de9be2be73409387480db617b0031c with peer 64688d2615e44408ab868cb531540a84
I20240502 02:02:02.575815 31292 raft_consensus.cc:1203] T 3475ca93c949435699b590bd97e3735c P 64688d2615e44408ab868cb531540a84 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 02:02:02.659118 31559 raft_consensus.cc:1203] T fecea55aa2ab478d91bcad93eca93aef P e325c0c7447f4b329994aafdee580666 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 02:02:02.733366 31292 raft_consensus.cc:1203] T 28de9be2be73409387480db617b0031c P 64688d2615e44408ab868cb531540a84 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 02:02:03.137723   690 raft_consensus.cc:1050] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb: attempting to promote NON_VOTER 64688d2615e44408ab868cb531540a84 to VOTER
I20240502 02:02:03.138726   690 consensus_queue.cc:237] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { promote: false } }
I20240502 02:02:03.141382 31292 raft_consensus.cc:1261] T 28de9be2be73409387480db617b0031c P 64688d2615e44408ab868cb531540a84 [term 1 LEARNER]: Refusing update from remote peer 0db0855dad46416094c68fc878712cbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 02:02:03.141382 32229 raft_consensus.cc:1261] T 28de9be2be73409387480db617b0031c P 29755ed6e9444b2daf5807bd23451d29 [term 1 FOLLOWER]: Refusing update from remote peer 0db0855dad46416094c68fc878712cbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 02:02:03.141449 31559 raft_consensus.cc:1261] T 28de9be2be73409387480db617b0031c P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Refusing update from remote peer 0db0855dad46416094c68fc878712cbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 02:02:03.142071   690 consensus_queue.cc:1035] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 02:02:03.142452   723 consensus_queue.cc:1035] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 02:02:03.142814   323 consensus_queue.cc:1035] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 02:02:03.146158   323 raft_consensus.cc:2932] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 64688d2615e44408ab868cb531540a84 (127.25.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { promote: false } } }
I20240502 02:02:03.146808   689 raft_consensus.cc:1050] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666: attempting to promote NON_VOTER 64688d2615e44408ab868cb531540a84 to VOTER
I20240502 02:02:03.147514   689 consensus_queue.cc:237] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { promote: false } }
I20240502 02:02:03.149984 32229 raft_consensus.cc:2932] T 28de9be2be73409387480db617b0031c P 29755ed6e9444b2daf5807bd23451d29 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 64688d2615e44408ab868cb531540a84 (127.25.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { promote: false } } }
I20240502 02:02:03.150468 32363 raft_consensus.cc:1261] T 3475ca93c949435699b590bd97e3735c P 7da2c0838fce40c5b5000c4d71c72708 [term 1 FOLLOWER]: Refusing update from remote peer e325c0c7447f4b329994aafdee580666: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 02:02:03.150580 31292 raft_consensus.cc:1261] T 3475ca93c949435699b590bd97e3735c P 64688d2615e44408ab868cb531540a84 [term 1 LEARNER]: Refusing update from remote peer e325c0c7447f4b329994aafdee580666: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 02:02:03.151146   727 consensus_queue.cc:1035] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 02:02:03.151852   729 consensus_queue.cc:1035] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 02:02:03.152340 31146 catalog_manager.cc:5484] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb reported cstate change: config changed from index 7 to 8, 64688d2615e44408ab868cb531540a84 (127.25.109.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0db0855dad46416094c68fc878712cbb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 02:02:03.154268 31559 raft_consensus.cc:2932] T 28de9be2be73409387480db617b0031c P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 64688d2615e44408ab868cb531540a84 (127.25.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { promote: false } } }
I20240502 02:02:03.154402 31291 raft_consensus.cc:2932] T 28de9be2be73409387480db617b0031c P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 64688d2615e44408ab868cb531540a84 (127.25.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { promote: false } } }
I20240502 02:02:03.160662 31426 raft_consensus.cc:1261] T 3475ca93c949435699b590bd97e3735c P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Refusing update from remote peer e325c0c7447f4b329994aafdee580666: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 02:02:03.161496   729 consensus_queue.cc:1035] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 02:02:03.166707   689 raft_consensus.cc:2932] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 64688d2615e44408ab868cb531540a84 (127.25.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { promote: false } } }
I20240502 02:02:03.167709 31292 raft_consensus.cc:2932] T 3475ca93c949435699b590bd97e3735c P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 64688d2615e44408ab868cb531540a84 (127.25.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { promote: false } } }
I20240502 02:02:03.168067 31426 raft_consensus.cc:2932] T 3475ca93c949435699b590bd97e3735c P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 64688d2615e44408ab868cb531540a84 (127.25.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { promote: false } } }
I20240502 02:02:03.171806 32363 raft_consensus.cc:2932] T 3475ca93c949435699b590bd97e3735c P 7da2c0838fce40c5b5000c4d71c72708 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 64688d2615e44408ab868cb531540a84 (127.25.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { promote: false } } }
I20240502 02:02:03.181331 31150 catalog_manager.cc:5484] T 3475ca93c949435699b590bd97e3735c P 0db0855dad46416094c68fc878712cbb reported cstate change: config changed from index 7 to 8, 64688d2615e44408ab868cb531540a84 (127.25.109.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e325c0c7447f4b329994aafdee580666" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } attrs { replace: true } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { promote: false } } }
I20240502 02:02:03.185904 31426 consensus_queue.cc:237] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { promote: false } }
I20240502 02:02:03.186497 31559 consensus_queue.cc:237] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { promote: false } }
I20240502 02:02:03.188547 32229 raft_consensus.cc:1261] T 28de9be2be73409387480db617b0031c P 29755ed6e9444b2daf5807bd23451d29 [term 1 FOLLOWER]: Refusing update from remote peer 0db0855dad46416094c68fc878712cbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 02:02:03.189124 31292 raft_consensus.cc:1261] T 3475ca93c949435699b590bd97e3735c P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Refusing update from remote peer e325c0c7447f4b329994aafdee580666: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 02:02:03.189158   323 consensus_queue.cc:1035] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 02:02:03.189373 31291 raft_consensus.cc:1261] T 28de9be2be73409387480db617b0031c P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Refusing update from remote peer 0db0855dad46416094c68fc878712cbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 02:02:03.188709 32363 raft_consensus.cc:1261] T 3475ca93c949435699b590bd97e3735c P 7da2c0838fce40c5b5000c4d71c72708 [term 1 FOLLOWER]: Refusing update from remote peer e325c0c7447f4b329994aafdee580666: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 02:02:03.190171   323 consensus_queue.cc:1035] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 02:02:03.190254   734 consensus_queue.cc:1035] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 02:02:03.190645   729 consensus_queue.cc:1035] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 02:02:03.192049   323 raft_consensus.cc:2932] T 28de9be2be73409387480db617b0031c P 0db0855dad46416094c68fc878712cbb [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e325c0c7447f4b329994aafdee580666 (127.25.109.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { promote: false } } }
I20240502 02:02:03.192453 32229 raft_consensus.cc:2932] T 28de9be2be73409387480db617b0031c P 29755ed6e9444b2daf5807bd23451d29 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e325c0c7447f4b329994aafdee580666 (127.25.109.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { promote: false } } }
I20240502 02:02:03.196564 31137 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 28de9be2be73409387480db617b0031c with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 02:02:03.196319 31291 raft_consensus.cc:2932] T 28de9be2be73409387480db617b0031c P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e325c0c7447f4b329994aafdee580666 (127.25.109.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { promote: false } } }
I20240502 02:02:03.197018 31141 catalog_manager.cc:5484] T 28de9be2be73409387480db617b0031c P 29755ed6e9444b2daf5807bd23451d29 reported cstate change: config changed from index 8 to 9, VOTER e325c0c7447f4b329994aafdee580666 (127.25.109.131) evicted. New cstate: current_term: 1 leader_uuid: "0db0855dad46416094c68fc878712cbb" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { promote: false } } }
I20240502 02:02:03.198364   727 raft_consensus.cc:2932] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 0db0855dad46416094c68fc878712cbb (127.25.109.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { promote: false } } }
I20240502 02:02:03.200162   690 raft_consensus.cc:1050] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb: attempting to promote NON_VOTER e325c0c7447f4b329994aafdee580666 to VOTER
I20240502 02:02:03.199820 31292 raft_consensus.cc:2932] T 3475ca93c949435699b590bd97e3735c P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 0db0855dad46416094c68fc878712cbb (127.25.109.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { promote: false } } }
I20240502 02:02:03.201025   690 consensus_queue.cc:237] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { promote: false } }
I20240502 02:02:03.203625 32363 raft_consensus.cc:2932] T 3475ca93c949435699b590bd97e3735c P 7da2c0838fce40c5b5000c4d71c72708 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 0db0855dad46416094c68fc878712cbb (127.25.109.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { promote: false } } }
I20240502 02:02:03.204063 31136 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3475ca93c949435699b590bd97e3735c with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 02:02:03.206064 31150 catalog_manager.cc:5484] T 3475ca93c949435699b590bd97e3735c P e325c0c7447f4b329994aafdee580666 reported cstate change: config changed from index 8 to 9, VOTER 0db0855dad46416094c68fc878712cbb (127.25.109.130) evicted. New cstate: current_term: 1 leader_uuid: "e325c0c7447f4b329994aafdee580666" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7da2c0838fce40c5b5000c4d71c72708" member_type: VOTER last_known_addr { host: "127.25.109.133" port: 36143 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 02:02:03.207434 32229 raft_consensus.cc:1261] T fecea55aa2ab478d91bcad93eca93aef P 29755ed6e9444b2daf5807bd23451d29 [term 1 FOLLOWER]: Refusing update from remote peer 0db0855dad46416094c68fc878712cbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 02:02:03.208125   690 consensus_queue.cc:1035] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 02:02:03.210028 31538 tablet_service.cc:1514] Processing DeleteTablet for tablet 28de9be2be73409387480db617b0031c with delete_type TABLET_DATA_TOMBSTONED (TS e325c0c7447f4b329994aafdee580666 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:59068
I20240502 02:02:03.210834 31559 raft_consensus.cc:1261] T fecea55aa2ab478d91bcad93eca93aef P e325c0c7447f4b329994aafdee580666 [term 1 LEARNER]: Refusing update from remote peer 0db0855dad46416094c68fc878712cbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 02:02:03.211783 31291 raft_consensus.cc:1261] T fecea55aa2ab478d91bcad93eca93aef P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Refusing update from remote peer 0db0855dad46416094c68fc878712cbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 02:02:03.212303   690 consensus_queue.cc:1035] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 02:02:03.212728   324 consensus_queue.cc:1035] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 02:02:03.216637 31407 tablet_service.cc:1514] Processing DeleteTablet for tablet 3475ca93c949435699b590bd97e3735c with delete_type TABLET_DATA_TOMBSTONED (TS 0db0855dad46416094c68fc878712cbb not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:43004
I20240502 02:02:03.216491   323 raft_consensus.cc:2932] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e325c0c7447f4b329994aafdee580666 (127.25.109.131) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { promote: false } } }
I20240502 02:02:03.217919 31559 raft_consensus.cc:2932] T fecea55aa2ab478d91bcad93eca93aef P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e325c0c7447f4b329994aafdee580666 (127.25.109.131) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { promote: false } } }
I20240502 02:02:03.221462 32230 raft_consensus.cc:2932] T fecea55aa2ab478d91bcad93eca93aef P 29755ed6e9444b2daf5807bd23451d29 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e325c0c7447f4b329994aafdee580666 (127.25.109.131) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { promote: false } } }
I20240502 02:02:03.224195 31150 catalog_manager.cc:5484] T fecea55aa2ab478d91bcad93eca93aef P e325c0c7447f4b329994aafdee580666 reported cstate change: config changed from index 7 to 8, e325c0c7447f4b329994aafdee580666 (127.25.109.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0db0855dad46416094c68fc878712cbb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { promote: false } } }
I20240502 02:02:03.228533   742 tablet_replica.cc:330] T 28de9be2be73409387480db617b0031c P e325c0c7447f4b329994aafdee580666: stopping tablet replica
I20240502 02:02:03.228770   742 raft_consensus.cc:2229] T 28de9be2be73409387480db617b0031c P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 02:02:03.229221   745 tablet_replica.cc:330] T 3475ca93c949435699b590bd97e3735c P 0db0855dad46416094c68fc878712cbb: stopping tablet replica
I20240502 02:02:03.229261   742 raft_consensus.cc:2258] T 28de9be2be73409387480db617b0031c P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 02:02:03.229427   745 raft_consensus.cc:2229] T 3475ca93c949435699b590bd97e3735c P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 02:02:03.229779   745 raft_consensus.cc:2258] T 3475ca93c949435699b590bd97e3735c P 0db0855dad46416094c68fc878712cbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 02:02:03.231516   742 ts_tablet_manager.cc:1875] T 28de9be2be73409387480db617b0031c P e325c0c7447f4b329994aafdee580666: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 02:02:03.231647 31291 raft_consensus.cc:2932] T fecea55aa2ab478d91bcad93eca93aef P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e325c0c7447f4b329994aafdee580666 (127.25.109.131) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "64688d2615e44408ab868cb531540a84" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33021 } attrs { replace: true } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { promote: false } } }
I20240502 02:02:03.233225 31426 consensus_queue.cc:237] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { promote: false } }
I20240502 02:02:03.235357 32230 raft_consensus.cc:1261] T fecea55aa2ab478d91bcad93eca93aef P 29755ed6e9444b2daf5807bd23451d29 [term 1 FOLLOWER]: Refusing update from remote peer 0db0855dad46416094c68fc878712cbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 02:02:03.235962 31559 raft_consensus.cc:1261] T fecea55aa2ab478d91bcad93eca93aef P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Refusing update from remote peer 0db0855dad46416094c68fc878712cbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 02:02:03.236125   324 consensus_queue.cc:1035] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 02:02:03.237012   324 consensus_queue.cc:1035] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 02:02:03.240320   745 ts_tablet_manager.cc:1875] T 3475ca93c949435699b590bd97e3735c P 0db0855dad46416094c68fc878712cbb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 02:02:03.240373   324 raft_consensus.cc:2932] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 64688d2615e44408ab868cb531540a84 (127.25.109.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { promote: false } } }
I20240502 02:02:03.241362 31558 raft_consensus.cc:2932] T fecea55aa2ab478d91bcad93eca93aef P e325c0c7447f4b329994aafdee580666 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 64688d2615e44408ab868cb531540a84 (127.25.109.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { promote: false } } }
I20240502 02:02:03.245348 31137 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fecea55aa2ab478d91bcad93eca93aef with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 02:02:03.246802 31150 catalog_manager.cc:5484] T fecea55aa2ab478d91bcad93eca93aef P 0db0855dad46416094c68fc878712cbb reported cstate change: config changed from index 8 to 9, VOTER 64688d2615e44408ab868cb531540a84 (127.25.109.129) evicted. New cstate: current_term: 1 leader_uuid: "0db0855dad46416094c68fc878712cbb" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 02:02:03.248216 32230 raft_consensus.cc:2932] T fecea55aa2ab478d91bcad93eca93aef P 29755ed6e9444b2daf5807bd23451d29 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 64688d2615e44408ab868cb531540a84 (127.25.109.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0db0855dad46416094c68fc878712cbb" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 38025 } } peers { permanent_uuid: "29755ed6e9444b2daf5807bd23451d29" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 38023 } attrs { promote: false } } peers { permanent_uuid: "e325c0c7447f4b329994aafdee580666" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 43937 } attrs { promote: false } } }
I20240502 02:02:03.249338   745 ts_tablet_manager.cc:1888] T 3475ca93c949435699b590bd97e3735c P 0db0855dad46416094c68fc878712cbb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 02:02:03.249555   745 log.cc:1196] T 3475ca93c949435699b590bd97e3735c P 0db0855dad46416094c68fc878712cbb: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal/wals/3475ca93c949435699b590bd97e3735c
I20240502 02:02:03.250522 31137 catalog_manager.cc:4830] TS 0db0855dad46416094c68fc878712cbb (127.25.109.130:38025): tablet 3475ca93c949435699b590bd97e3735c (table rebalance_test_table_0 [id=c8cf2d1749644bbb8b2330bcbf7cb196]) successfully deleted
I20240502 02:02:03.252076 31270 tablet_service.cc:1514] Processing DeleteTablet for tablet fecea55aa2ab478d91bcad93eca93aef with delete_type TABLET_DATA_TOMBSTONED (TS 64688d2615e44408ab868cb531540a84 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:48614
I20240502 02:02:03.253039   748 tablet_replica.cc:330] T fecea55aa2ab478d91bcad93eca93aef P 64688d2615e44408ab868cb531540a84: stopping tablet replica
I20240502 02:02:03.253330   748 raft_consensus.cc:2229] T fecea55aa2ab478d91bcad93eca93aef P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 02:02:03.253902   748 raft_consensus.cc:2258] T fecea55aa2ab478d91bcad93eca93aef P 64688d2615e44408ab868cb531540a84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 02:02:03.254897   748 ts_tablet_manager.cc:1875] T fecea55aa2ab478d91bcad93eca93aef P 64688d2615e44408ab868cb531540a84: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 02:02:03.259924   748 ts_tablet_manager.cc:1888] T fecea55aa2ab478d91bcad93eca93aef P 64688d2615e44408ab868cb531540a84: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 02:02:03.260067   748 log.cc:1196] T fecea55aa2ab478d91bcad93eca93aef P 64688d2615e44408ab868cb531540a84: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal/wals/fecea55aa2ab478d91bcad93eca93aef
I20240502 02:02:03.260680   742 ts_tablet_manager.cc:1888] T 28de9be2be73409387480db617b0031c P e325c0c7447f4b329994aafdee580666: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 02:02:03.260849   742 log.cc:1196] T 28de9be2be73409387480db617b0031c P e325c0c7447f4b329994aafdee580666: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/wals/28de9be2be73409387480db617b0031c
I20240502 02:02:03.260838 31136 catalog_manager.cc:4830] TS 64688d2615e44408ab868cb531540a84 (127.25.109.129:33021): tablet fecea55aa2ab478d91bcad93eca93aef (table rebalance_test_table_1 [id=1e4e557617e14b7895769013935449f4]) successfully deleted
I20240502 02:02:03.261700 31136 catalog_manager.cc:4830] TS e325c0c7447f4b329994aafdee580666 (127.25.109.131:43937): tablet 28de9be2be73409387480db617b0031c (table rebalance_test_table_0 [id=c8cf2d1749644bbb8b2330bcbf7cb196]) successfully deleted
I20240502 02:02:03.377558 31407 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:02:03.403981 32343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:02:03.412127 31270 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:02:03.414821 31538 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:03.416005 32210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:02:03.568092 31407 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:02:03.581156 31270 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:02:03.588294 32343 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:02:03.588739 32210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:02:03.609408 31538 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 02:02:03.819564 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 31209
I20240502 02:02:03.839375 31333 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 02:02:03.981528 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 31209
W20240502 02:02:04.004566 31493 connection.cc:535] server connection from 127.25.109.129:40369 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 02:02:04.005067 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 31344
I20240502 02:02:04.011085 31467 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 02:02:04.164274 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 31344
I20240502 02:02:04.187274 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 31476
I20240502 02:02:04.193742 31601 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 02:02:04.333250 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 31476
I20240502 02:02:04.355959 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 32146
W20240502 02:02:04.355937 31136 connection.cc:535] server connection from 127.25.109.131:53025 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 02:02:04.362443 32269 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 02:02:04.476491 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 32146
I20240502 02:02:04.495651 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 32279
I20240502 02:02:04.502429 32404 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 02:02:04.614113 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 32279
I20240502 02:02:04.633059 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 31119
I20240502 02:02:04.634279 31180 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 02:02:04.780364 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 31119
2024-05-02T02:02:04Z chronyd exiting
[       OK ] LocationAwareRebalancingParamTest.Rebalance/idx2_rf3_t3_l_3_2 (18334 ms)
[ RUN      ] LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer/idx1_rf3_t3_l_3_1
I20240502 02:02:04.832998 26038 test_util.cc:268] Using random seed: -44239665
I20240502 02:02:04.834882 26038 ts_itest-base.cc:112] Starting cluster with:
I20240502 02:02:04.834944 26038 ts_itest-base.cc:113] --------------
I20240502 02:02:04.834986 26038 ts_itest-base.cc:114] 4 tablet servers
I20240502 02:02:04.835026 26038 ts_itest-base.cc:115] 3 replicas per TS
I20240502 02:02:04.835072 26038 ts_itest-base.cc:116] --------------
2024-05-02T02:02:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T02:02:04Z Disabled control of system clock
I20240502 02:02:04.892434 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.190:33343
--webserver_interface=127.25.109.190
--webserver_port=0
--builtin_ntp_servers=127.25.109.148:45821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.109.190:33343
--location_mapping_cmd=/tmp/dist-test-taskb4pM1q/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:1
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240502 02:02:05.229194   844 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 02:02:05.229635   844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:02:05.229727   844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 02:02:05.229805   844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:02:05.237787   844 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 02:02:05.237932   844 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 02:02:05.238010   844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 02:02:05.238058   844 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 02:02:05.238118   844 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 02:02:05.247627   844 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:45821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskb4pM1q/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:1
--ipki_ca_key_size=768
--master_addresses=127.25.109.190:33343
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.109.190:33343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.25.109.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 02:02:05.249048   844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 02:02:05.250753   844 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 02:02:05.261910   852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:02:05.261966   850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:02:05.262184   849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 02:02:05.262784   844 server_base.cc:1034] running on GCE node
I20240502 02:02:05.263823   844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 02:02:05.265303   844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 02:02:05.266530   844 hybrid_clock.cc:648] HybridClock initialized: now 1714615325266455 us; error 44 us; skew 500 ppm
I20240502 02:02:05.266846   844 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 02:02:05.269182   844 webserver.cc:460] Webserver started at http://127.25.109.190:43209/ using document root <none> and password file <none>
I20240502 02:02:05.269928   844 fs_manager.cc:359] Metadata directory not provided
I20240502 02:02:05.270068   844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 02:02:05.270367   844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 02:02:05.273133   844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "d3265b3a09714fa18506aa022236491d"
format_stamp: "Formatted at 2024-05-02 02:02:05 on dist-test-slave-kj4b"
I20240502 02:02:05.273923   844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "d3265b3a09714fa18506aa022236491d"
format_stamp: "Formatted at 2024-05-02 02:02:05 on dist-test-slave-kj4b"
I20240502 02:02:05.280215   844 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.002s	sys 0.005s
I20240502 02:02:05.283763   858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:02:05.285188   844 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 02:02:05.285368   844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal
uuid: "d3265b3a09714fa18506aa022236491d"
format_stamp: "Formatted at 2024-05-02 02:02:05 on dist-test-slave-kj4b"
I20240502 02:02:05.285564   844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 02:02:05.338872   844 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 02:02:05.339838   844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 02:02:05.340178   844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 02:02:05.358278   844 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.190:33343
I20240502 02:02:05.358318   910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.190:33343 every 8 connection(s)
I20240502 02:02:05.359679   844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 02:02:05.363478   911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:05.365789 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 844
I20240502 02:02:05.365995 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 02:02:05.374447   911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3265b3a09714fa18506aa022236491d: Bootstrap starting.
I20240502 02:02:05.377699   911 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3265b3a09714fa18506aa022236491d: Neither blocks nor log segments found. Creating new log.
I20240502 02:02:05.379026   911 log.cc:824] T 00000000000000000000000000000000 P d3265b3a09714fa18506aa022236491d: Log is configured to *not* fsync() on all Append() calls
I20240502 02:02:05.381881   911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3265b3a09714fa18506aa022236491d: No bootstrap required, opened a new log
I20240502 02:02:05.388236   911 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3265b3a09714fa18506aa022236491d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3265b3a09714fa18506aa022236491d" member_type: VOTER last_known_addr { host: "127.25.109.190" port: 33343 } }
I20240502 02:02:05.388696   911 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3265b3a09714fa18506aa022236491d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:02:05.388824   911 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3265b3a09714fa18506aa022236491d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3265b3a09714fa18506aa022236491d, State: Initialized, Role: FOLLOWER
I20240502 02:02:05.389461   911 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3265b3a09714fa18506aa022236491d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3265b3a09714fa18506aa022236491d" member_type: VOTER last_known_addr { host: "127.25.109.190" port: 33343 } }
I20240502 02:02:05.389655   911 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3265b3a09714fa18506aa022236491d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 02:02:05.389755   911 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3265b3a09714fa18506aa022236491d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 02:02:05.389887   911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3265b3a09714fa18506aa022236491d [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:02:05.392015   911 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3265b3a09714fa18506aa022236491d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3265b3a09714fa18506aa022236491d" member_type: VOTER last_known_addr { host: "127.25.109.190" port: 33343 } }
I20240502 02:02:05.392616   911 leader_election.cc:302] T 00000000000000000000000000000000 P d3265b3a09714fa18506aa022236491d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3265b3a09714fa18506aa022236491d; no voters: 
I20240502 02:02:05.393301   911 leader_election.cc:288] T 00000000000000000000000000000000 P d3265b3a09714fa18506aa022236491d [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 02:02:05.393438   916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3265b3a09714fa18506aa022236491d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:02:05.394047   916 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3265b3a09714fa18506aa022236491d [term 1 LEADER]: Becoming Leader. State: Replica: d3265b3a09714fa18506aa022236491d, State: Running, Role: LEADER
I20240502 02:02:05.394616   916 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3265b3a09714fa18506aa022236491d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3265b3a09714fa18506aa022236491d" member_type: VOTER last_known_addr { host: "127.25.109.190" port: 33343 } }
I20240502 02:02:05.395339   911 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3265b3a09714fa18506aa022236491d [sys.catalog]: configured and running, proceeding with master startup.
I20240502 02:02:05.397239   917 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3265b3a09714fa18506aa022236491d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3265b3a09714fa18506aa022236491d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3265b3a09714fa18506aa022236491d" member_type: VOTER last_known_addr { host: "127.25.109.190" port: 33343 } } }
I20240502 02:02:05.397519   917 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3265b3a09714fa18506aa022236491d [sys.catalog]: This master's current role is: LEADER
I20240502 02:02:05.398116   919 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3265b3a09714fa18506aa022236491d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3265b3a09714fa18506aa022236491d. Latest consensus state: current_term: 1 leader_uuid: "d3265b3a09714fa18506aa022236491d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3265b3a09714fa18506aa022236491d" member_type: VOTER last_known_addr { host: "127.25.109.190" port: 33343 } } }
I20240502 02:02:05.398392   919 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3265b3a09714fa18506aa022236491d [sys.catalog]: This master's current role is: LEADER
I20240502 02:02:05.409654   923 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 02:02:05.414108   923 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 02:02:05.422768   923 catalog_manager.cc:1347] Generated new cluster ID: 8f5756ddafc548328c9e3cce3334d7f9
I20240502 02:02:05.422910   923 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 02:02:05.446338   923 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 02:02:05.447566   923 catalog_manager.cc:1504] Loading token signing keys...
I20240502 02:02:05.456427   923 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d3265b3a09714fa18506aa022236491d: Generated new TSK 0
I20240502 02:02:05.457607   923 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 02:02:05.475720 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.129:0
--local_ip_for_outbound_sockets=127.25.109.129
--webserver_interface=127.25.109.129
--webserver_port=0
--tserver_master_addrs=127.25.109.190:33343
--builtin_ntp_servers=127.25.109.148:45821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 02:02:05.824980   936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:02:05.825275   936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 02:02:05.825381   936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:02:05.833093   936 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 02:02:05.833226   936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 02:02:05.833348   936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.129
I20240502 02:02:05.842612   936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:45821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.25.109.129
--webserver_port=0
--tserver_master_addrs=127.25.109.190:33343
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.129
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 02:02:05.844065   936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 02:02:05.845855   936 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 02:02:05.857570   941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:02:05.858042   942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:02:05.858340   944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 02:02:05.858429   936 server_base.cc:1034] running on GCE node
I20240502 02:02:05.859031   936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 02:02:05.860006   936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 02:02:05.861232   936 hybrid_clock.cc:648] HybridClock initialized: now 1714615325861182 us; error 40 us; skew 500 ppm
I20240502 02:02:05.861574   936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 02:02:05.863840   936 webserver.cc:460] Webserver started at http://127.25.109.129:42919/ using document root <none> and password file <none>
I20240502 02:02:05.864629   936 fs_manager.cc:359] Metadata directory not provided
I20240502 02:02:05.864743   936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 02:02:05.865051   936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 02:02:05.867678   936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "d0e4135c07ac4f2389ffe4e7379df71f"
format_stamp: "Formatted at 2024-05-02 02:02:05 on dist-test-slave-kj4b"
I20240502 02:02:05.868455   936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "d0e4135c07ac4f2389ffe4e7379df71f"
format_stamp: "Formatted at 2024-05-02 02:02:05 on dist-test-slave-kj4b"
I20240502 02:02:05.873440   936 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20240502 02:02:05.876842   950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:02:05.878140   936 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240502 02:02:05.878290   936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "d0e4135c07ac4f2389ffe4e7379df71f"
format_stamp: "Formatted at 2024-05-02 02:02:05 on dist-test-slave-kj4b"
I20240502 02:02:05.878505   936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 02:02:05.890686   936 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 02:02:05.891492   936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 02:02:05.891746   936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 02:02:05.892740   936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 02:02:05.894213   936 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 02:02:05.894307   936 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:02:05.894415   936 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 02:02:05.894483   936 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:02:05.924726   936 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.129:33963
I20240502 02:02:05.924785  1065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.129:33963 every 8 connection(s)
I20240502 02:02:05.926031   936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 02:02:05.930473 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 936
I20240502 02:02:05.930683 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 02:02:05.934161 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.130:0
--local_ip_for_outbound_sockets=127.25.109.130
--webserver_interface=127.25.109.130
--webserver_port=0
--tserver_master_addrs=127.25.109.190:33343
--builtin_ntp_servers=127.25.109.148:45821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 02:02:05.941848  1066 heartbeater.cc:347] Connected to a master server at 127.25.109.190:33343
I20240502 02:02:05.942193  1066 heartbeater.cc:464] Registering TS with master...
I20240502 02:02:05.943140  1066 heartbeater.cc:511] Master 127.25.109.190:33343 requested a full tablet report, sending...
I20240502 02:02:05.990154   875 ts_manager.cc:194] Registered new tserver with Master: d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129:33963)
I20240502 02:02:05.992091   875 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.129:40615
W20240502 02:02:06.281646  1070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:02:06.282016  1070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 02:02:06.282153  1070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:02:06.290123  1070 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 02:02:06.290247  1070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 02:02:06.290376  1070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.130
I20240502 02:02:06.299507  1070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:45821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.25.109.130
--webserver_port=0
--tserver_master_addrs=127.25.109.190:33343
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.130
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 02:02:06.300837  1070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 02:02:06.302549  1070 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 02:02:06.314025  1076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:02:06.314083  1077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:02:06.314270  1079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 02:02:06.314752  1070 server_base.cc:1034] running on GCE node
I20240502 02:02:06.315274  1070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 02:02:06.316192  1070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 02:02:06.317385  1070 hybrid_clock.cc:648] HybridClock initialized: now 1714615326317331 us; error 48 us; skew 500 ppm
I20240502 02:02:06.317693  1070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 02:02:06.319731  1070 webserver.cc:460] Webserver started at http://127.25.109.130:37945/ using document root <none> and password file <none>
I20240502 02:02:06.320451  1070 fs_manager.cc:359] Metadata directory not provided
I20240502 02:02:06.320564  1070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 02:02:06.320868  1070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 02:02:06.323426  1070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "8b9e878e72f64482bdd89f28ac427494"
format_stamp: "Formatted at 2024-05-02 02:02:06 on dist-test-slave-kj4b"
I20240502 02:02:06.324255  1070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "8b9e878e72f64482bdd89f28ac427494"
format_stamp: "Formatted at 2024-05-02 02:02:06 on dist-test-slave-kj4b"
I20240502 02:02:06.329159  1070 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.000s	sys 0.004s
I20240502 02:02:06.332459  1085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:02:06.333803  1070 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240502 02:02:06.333947  1070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "8b9e878e72f64482bdd89f28ac427494"
format_stamp: "Formatted at 2024-05-02 02:02:06 on dist-test-slave-kj4b"
I20240502 02:02:06.334146  1070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 02:02:06.357903  1070 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 02:02:06.358976  1070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 02:02:06.359306  1070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 02:02:06.360448  1070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 02:02:06.362363  1070 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 02:02:06.362478  1070 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:02:06.362601  1070 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 02:02:06.362680  1070 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:02:06.393775  1070 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.130:43631
I20240502 02:02:06.393828  1200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.130:43631 every 8 connection(s)
I20240502 02:02:06.395067  1070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 02:02:06.398689 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 1070
I20240502 02:02:06.398862 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 02:02:06.402549 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.131:0
--local_ip_for_outbound_sockets=127.25.109.131
--webserver_interface=127.25.109.131
--webserver_port=0
--tserver_master_addrs=127.25.109.190:33343
--builtin_ntp_servers=127.25.109.148:45821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 02:02:06.408386  1201 heartbeater.cc:347] Connected to a master server at 127.25.109.190:33343
I20240502 02:02:06.408692  1201 heartbeater.cc:464] Registering TS with master...
I20240502 02:02:06.409507  1201 heartbeater.cc:511] Master 127.25.109.190:33343 requested a full tablet report, sending...
I20240502 02:02:06.452077   875 ts_manager.cc:194] Registered new tserver with Master: 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130:43631)
I20240502 02:02:06.453023   875 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.130:52735
W20240502 02:02:06.733695  1204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:02:06.733978  1204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 02:02:06.734112  1204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:02:06.741353  1204 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 02:02:06.741469  1204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 02:02:06.741626  1204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.131
I20240502 02:02:06.750360  1204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:45821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.25.109.131
--webserver_port=0
--tserver_master_addrs=127.25.109.190:33343
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.131
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 02:02:06.751610  1204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 02:02:06.753283  1204 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 02:02:06.764436  1210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:02:06.764900  1211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:02:06.765502  1213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 02:02:06.765575  1204 server_base.cc:1034] running on GCE node
I20240502 02:02:06.766252  1204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 02:02:06.767235  1204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 02:02:06.768435  1204 hybrid_clock.cc:648] HybridClock initialized: now 1714615326768386 us; error 49 us; skew 500 ppm
I20240502 02:02:06.768785  1204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 02:02:06.771356  1204 webserver.cc:460] Webserver started at http://127.25.109.131:45291/ using document root <none> and password file <none>
I20240502 02:02:06.772269  1204 fs_manager.cc:359] Metadata directory not provided
I20240502 02:02:06.772404  1204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 02:02:06.772794  1204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 02:02:06.776257  1204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "d37e50431f30463c872c5d819ddf2c25"
format_stamp: "Formatted at 2024-05-02 02:02:06 on dist-test-slave-kj4b"
I20240502 02:02:06.777053  1204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "d37e50431f30463c872c5d819ddf2c25"
format_stamp: "Formatted at 2024-05-02 02:02:06 on dist-test-slave-kj4b"
I20240502 02:02:06.782101  1204 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.001s	sys 0.004s
I20240502 02:02:06.785423  1219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:02:06.786652  1204 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20240502 02:02:06.786784  1204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "d37e50431f30463c872c5d819ddf2c25"
format_stamp: "Formatted at 2024-05-02 02:02:06 on dist-test-slave-kj4b"
I20240502 02:02:06.786962  1204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 02:02:06.812752  1204 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 02:02:06.813548  1204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 02:02:06.813812  1204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 02:02:06.814704  1204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 02:02:06.816176  1204 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 02:02:06.816262  1204 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:02:06.816342  1204 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 02:02:06.816395  1204 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:02:06.844370  1204 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.131:37799
I20240502 02:02:06.844444  1333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.131:37799 every 8 connection(s)
I20240502 02:02:06.845659  1204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 02:02:06.846776 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 1204
I20240502 02:02:06.846987 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 02:02:06.850829 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.132:0
--local_ip_for_outbound_sockets=127.25.109.132
--webserver_interface=127.25.109.132
--webserver_port=0
--tserver_master_addrs=127.25.109.190:33343
--builtin_ntp_servers=127.25.109.148:45821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 02:02:06.859105  1334 heartbeater.cc:347] Connected to a master server at 127.25.109.190:33343
I20240502 02:02:06.859440  1334 heartbeater.cc:464] Registering TS with master...
I20240502 02:02:06.860445  1334 heartbeater.cc:511] Master 127.25.109.190:33343 requested a full tablet report, sending...
I20240502 02:02:06.903501   875 ts_manager.cc:194] Registered new tserver with Master: d37e50431f30463c872c5d819ddf2c25 (127.25.109.131:37799)
I20240502 02:02:06.904481   875 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.131:35115
I20240502 02:02:06.995802  1066 heartbeater.cc:503] Master 127.25.109.190:33343 was elected leader, sending a full tablet report...
W20240502 02:02:07.199185  1337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:02:07.199524  1337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 02:02:07.199672  1337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:02:07.207437  1337 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 02:02:07.207607  1337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 02:02:07.207767  1337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.132
I20240502 02:02:07.217124  1337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:45821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.25.109.132
--webserver_port=0
--tserver_master_addrs=127.25.109.190:33343
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.132
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 02:02:07.218487  1337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 02:02:07.220332  1337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 02:02:07.232774  1346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:02:07.232975  1344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:02:07.233561  1343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 02:02:07.233914  1337 server_base.cc:1034] running on GCE node
I20240502 02:02:07.234597  1337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 02:02:07.235615  1337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 02:02:07.236831  1337 hybrid_clock.cc:648] HybridClock initialized: now 1714615327236756 us; error 63 us; skew 500 ppm
I20240502 02:02:07.237169  1337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 02:02:07.239697  1337 webserver.cc:460] Webserver started at http://127.25.109.132:34929/ using document root <none> and password file <none>
I20240502 02:02:07.240521  1337 fs_manager.cc:359] Metadata directory not provided
I20240502 02:02:07.240630  1337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 02:02:07.240916  1337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 02:02:07.243626  1337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "eb5a2343514740caa85ff0228a535123"
format_stamp: "Formatted at 2024-05-02 02:02:07 on dist-test-slave-kj4b"
I20240502 02:02:07.244477  1337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "eb5a2343514740caa85ff0228a535123"
format_stamp: "Formatted at 2024-05-02 02:02:07 on dist-test-slave-kj4b"
I20240502 02:02:07.249992  1337 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240502 02:02:07.253675  1352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:02:07.255138  1337 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20240502 02:02:07.255308  1337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "eb5a2343514740caa85ff0228a535123"
format_stamp: "Formatted at 2024-05-02 02:02:07 on dist-test-slave-kj4b"
I20240502 02:02:07.255527  1337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 02:02:07.275923  1337 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 02:02:07.276872  1337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 02:02:07.277153  1337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 02:02:07.278170  1337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 02:02:07.279707  1337 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 02:02:07.279804  1337 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:02:07.279932  1337 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 02:02:07.280004  1337 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:02:07.312422  1337 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.132:44441
I20240502 02:02:07.312480  1467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.132:44441 every 8 connection(s)
I20240502 02:02:07.313791  1337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 02:02:07.315471 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 1337
I20240502 02:02:07.315642 26038 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 02:02:07.326665  1468 heartbeater.cc:347] Connected to a master server at 127.25.109.190:33343
I20240502 02:02:07.326994  1468 heartbeater.cc:464] Registering TS with master...
I20240502 02:02:07.327919  1468 heartbeater.cc:511] Master 127.25.109.190:33343 requested a full tablet report, sending...
I20240502 02:02:07.376947   875 ts_manager.cc:194] Registered new tserver with Master: eb5a2343514740caa85ff0228a535123 (127.25.109.132:44441)
I20240502 02:02:07.377915   875 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.132:51407
I20240502 02:02:07.388288 26038 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240502 02:02:07.406292 26038 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240502 02:02:07.406452 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 1337
I20240502 02:02:07.414901  1463 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 02:02:07.455714  1201 heartbeater.cc:503] Master 127.25.109.190:33343 was elected leader, sending a full tablet report...
I20240502 02:02:07.495190 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 1337
I20240502 02:02:07.907557  1334 heartbeater.cc:503] Master 127.25.109.190:33343 was elected leader, sending a full tablet report...
I20240502 02:02:11.262216   875 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42822:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240502 02:02:11.269502   875 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 02:02:11.308876   998 tablet_service.cc:1467] Processing CreateTablet for tablet b80d547302ec453aaf80fa6b64eebc13 (DEFAULT_TABLE table=rebalance_test_table_0 [id=3920dc53a6a846338bdd42178520e7af]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 02:02:11.308880  1134 tablet_service.cc:1467] Processing CreateTablet for tablet 9e7759be006b4a8186aca434c48949b2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=3920dc53a6a846338bdd42178520e7af]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 02:02:11.309870   997 tablet_service.cc:1467] Processing CreateTablet for tablet 9e7759be006b4a8186aca434c48949b2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=3920dc53a6a846338bdd42178520e7af]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 02:02:11.310627  1134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e7759be006b4a8186aca434c48949b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:11.310627   998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b80d547302ec453aaf80fa6b64eebc13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:11.310596   999 tablet_service.cc:1467] Processing CreateTablet for tablet 2a1ce57d725948239c211b4200d5f037 (DEFAULT_TABLE table=rebalance_test_table_0 [id=3920dc53a6a846338bdd42178520e7af]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 02:02:11.309062  1135 tablet_service.cc:1467] Processing CreateTablet for tablet b80d547302ec453aaf80fa6b64eebc13 (DEFAULT_TABLE table=rebalance_test_table_0 [id=3920dc53a6a846338bdd42178520e7af]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 02:02:11.311684   997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e7759be006b4a8186aca434c48949b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:11.311985   999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a1ce57d725948239c211b4200d5f037. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:11.314316  1267 tablet_service.cc:1467] Processing CreateTablet for tablet 9e7759be006b4a8186aca434c48949b2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=3920dc53a6a846338bdd42178520e7af]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 02:02:11.314358  1268 tablet_service.cc:1467] Processing CreateTablet for tablet b80d547302ec453aaf80fa6b64eebc13 (DEFAULT_TABLE table=rebalance_test_table_0 [id=3920dc53a6a846338bdd42178520e7af]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 02:02:11.316035  1269 tablet_service.cc:1467] Processing CreateTablet for tablet 2a1ce57d725948239c211b4200d5f037 (DEFAULT_TABLE table=rebalance_test_table_0 [id=3920dc53a6a846338bdd42178520e7af]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 02:02:11.316201  1268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b80d547302ec453aaf80fa6b64eebc13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:11.320108  1267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e7759be006b4a8186aca434c48949b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:11.320432  1135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b80d547302ec453aaf80fa6b64eebc13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:11.321136  1136 tablet_service.cc:1467] Processing CreateTablet for tablet 2a1ce57d725948239c211b4200d5f037 (DEFAULT_TABLE table=rebalance_test_table_0 [id=3920dc53a6a846338bdd42178520e7af]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 02:02:11.321695  1136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a1ce57d725948239c211b4200d5f037. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:11.322405  1269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a1ce57d725948239c211b4200d5f037. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:11.346117  1491 tablet_bootstrap.cc:492] T 9e7759be006b4a8186aca434c48949b2 P d0e4135c07ac4f2389ffe4e7379df71f: Bootstrap starting.
I20240502 02:02:11.348157  1492 tablet_bootstrap.cc:492] T 9e7759be006b4a8186aca434c48949b2 P d37e50431f30463c872c5d819ddf2c25: Bootstrap starting.
I20240502 02:02:11.349079  1493 tablet_bootstrap.cc:492] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494: Bootstrap starting.
I20240502 02:02:11.350849  1491 tablet_bootstrap.cc:654] T 9e7759be006b4a8186aca434c48949b2 P d0e4135c07ac4f2389ffe4e7379df71f: Neither blocks nor log segments found. Creating new log.
I20240502 02:02:11.351814  1492 tablet_bootstrap.cc:654] T 9e7759be006b4a8186aca434c48949b2 P d37e50431f30463c872c5d819ddf2c25: Neither blocks nor log segments found. Creating new log.
I20240502 02:02:11.351838  1493 tablet_bootstrap.cc:654] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494: Neither blocks nor log segments found. Creating new log.
I20240502 02:02:11.352339  1491 log.cc:824] T 9e7759be006b4a8186aca434c48949b2 P d0e4135c07ac4f2389ffe4e7379df71f: Log is configured to *not* fsync() on all Append() calls
I20240502 02:02:11.353034  1493 log.cc:824] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494: Log is configured to *not* fsync() on all Append() calls
I20240502 02:02:11.353140  1492 log.cc:824] T 9e7759be006b4a8186aca434c48949b2 P d37e50431f30463c872c5d819ddf2c25: Log is configured to *not* fsync() on all Append() calls
I20240502 02:02:11.355717  1491 tablet_bootstrap.cc:492] T 9e7759be006b4a8186aca434c48949b2 P d0e4135c07ac4f2389ffe4e7379df71f: No bootstrap required, opened a new log
I20240502 02:02:11.355967  1493 tablet_bootstrap.cc:492] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494: No bootstrap required, opened a new log
I20240502 02:02:11.356091  1492 tablet_bootstrap.cc:492] T 9e7759be006b4a8186aca434c48949b2 P d37e50431f30463c872c5d819ddf2c25: No bootstrap required, opened a new log
I20240502 02:02:11.356089  1491 ts_tablet_manager.cc:1367] T 9e7759be006b4a8186aca434c48949b2 P d0e4135c07ac4f2389ffe4e7379df71f: Time spent bootstrapping tablet: real 0.023s	user 0.006s	sys 0.004s
I20240502 02:02:11.356277  1493 ts_tablet_manager.cc:1367] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494: Time spent bootstrapping tablet: real 0.007s	user 0.002s	sys 0.005s
I20240502 02:02:11.356488  1492 ts_tablet_manager.cc:1367] T 9e7759be006b4a8186aca434c48949b2 P d37e50431f30463c872c5d819ddf2c25: Time spent bootstrapping tablet: real 0.009s	user 0.008s	sys 0.000s
I20240502 02:02:11.365532  1492 raft_consensus.cc:348] T 9e7759be006b4a8186aca434c48949b2 P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:11.365526  1491 raft_consensus.cc:348] T 9e7759be006b4a8186aca434c48949b2 P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:11.365526  1493 raft_consensus.cc:348] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:11.366053  1491 raft_consensus.cc:374] T 9e7759be006b4a8186aca434c48949b2 P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:02:11.366058  1493 raft_consensus.cc:374] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:02:11.366160  1491 raft_consensus.cc:729] T 9e7759be006b4a8186aca434c48949b2 P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0e4135c07ac4f2389ffe4e7379df71f, State: Initialized, Role: FOLLOWER
I20240502 02:02:11.366160  1493 raft_consensus.cc:729] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b9e878e72f64482bdd89f28ac427494, State: Initialized, Role: FOLLOWER
I20240502 02:02:11.366358  1492 raft_consensus.cc:374] T 9e7759be006b4a8186aca434c48949b2 P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:02:11.366523  1492 raft_consensus.cc:729] T 9e7759be006b4a8186aca434c48949b2 P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d37e50431f30463c872c5d819ddf2c25, State: Initialized, Role: FOLLOWER
I20240502 02:02:11.366999  1491 consensus_queue.cc:260] T 9e7759be006b4a8186aca434c48949b2 P d0e4135c07ac4f2389ffe4e7379df71f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:11.367594  1492 consensus_queue.cc:260] T 9e7759be006b4a8186aca434c48949b2 P d37e50431f30463c872c5d819ddf2c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:11.385653  1491 ts_tablet_manager.cc:1398] T 9e7759be006b4a8186aca434c48949b2 P d0e4135c07ac4f2389ffe4e7379df71f: Time spent starting tablet: real 0.029s	user 0.013s	sys 0.015s
I20240502 02:02:11.386132  1492 ts_tablet_manager.cc:1398] T 9e7759be006b4a8186aca434c48949b2 P d37e50431f30463c872c5d819ddf2c25: Time spent starting tablet: real 0.029s	user 0.016s	sys 0.012s
I20240502 02:02:11.386209  1491 tablet_bootstrap.cc:492] T 2a1ce57d725948239c211b4200d5f037 P d0e4135c07ac4f2389ffe4e7379df71f: Bootstrap starting.
I20240502 02:02:11.386776  1492 tablet_bootstrap.cc:492] T b80d547302ec453aaf80fa6b64eebc13 P d37e50431f30463c872c5d819ddf2c25: Bootstrap starting.
I20240502 02:02:11.386871  1493 consensus_queue.cc:260] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:11.388298  1491 tablet_bootstrap.cc:654] T 2a1ce57d725948239c211b4200d5f037 P d0e4135c07ac4f2389ffe4e7379df71f: Neither blocks nor log segments found. Creating new log.
I20240502 02:02:11.388877  1492 tablet_bootstrap.cc:654] T b80d547302ec453aaf80fa6b64eebc13 P d37e50431f30463c872c5d819ddf2c25: Neither blocks nor log segments found. Creating new log.
I20240502 02:02:11.412000  1497 raft_consensus.cc:482] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 02:02:11.412451  1497 raft_consensus.cc:504] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
W20240502 02:02:11.432070  1067 tablet.cc:2369] T 9e7759be006b4a8186aca434c48949b2 P d0e4135c07ac4f2389ffe4e7379df71f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 02:02:11.442095  1497 leader_election.cc:288] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129:33963), d37e50431f30463c872c5d819ddf2c25 (127.25.109.131:37799)
I20240502 02:02:11.465974  1493 ts_tablet_manager.cc:1398] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494: Time spent starting tablet: real 0.110s	user 0.045s	sys 0.058s
I20240502 02:02:11.466210  1491 tablet_bootstrap.cc:492] T 2a1ce57d725948239c211b4200d5f037 P d0e4135c07ac4f2389ffe4e7379df71f: No bootstrap required, opened a new log
I20240502 02:02:11.466434  1491 ts_tablet_manager.cc:1367] T 2a1ce57d725948239c211b4200d5f037 P d0e4135c07ac4f2389ffe4e7379df71f: Time spent bootstrapping tablet: real 0.080s	user 0.028s	sys 0.048s
I20240502 02:02:11.466594  1493 tablet_bootstrap.cc:492] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494: Bootstrap starting.
I20240502 02:02:11.467142  1492 tablet_bootstrap.cc:492] T b80d547302ec453aaf80fa6b64eebc13 P d37e50431f30463c872c5d819ddf2c25: No bootstrap required, opened a new log
I20240502 02:02:11.467383  1492 ts_tablet_manager.cc:1367] T b80d547302ec453aaf80fa6b64eebc13 P d37e50431f30463c872c5d819ddf2c25: Time spent bootstrapping tablet: real 0.081s	user 0.033s	sys 0.046s
I20240502 02:02:11.467607  1491 raft_consensus.cc:348] T 2a1ce57d725948239c211b4200d5f037 P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } }
I20240502 02:02:11.467909  1491 raft_consensus.cc:374] T 2a1ce57d725948239c211b4200d5f037 P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:02:11.468036  1491 raft_consensus.cc:729] T 2a1ce57d725948239c211b4200d5f037 P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0e4135c07ac4f2389ffe4e7379df71f, State: Initialized, Role: FOLLOWER
I20240502 02:02:11.468408  1493 tablet_bootstrap.cc:654] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494: Neither blocks nor log segments found. Creating new log.
I20240502 02:02:11.468392  1491 consensus_queue.cc:260] T 2a1ce57d725948239c211b4200d5f037 P d0e4135c07ac4f2389ffe4e7379df71f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } }
I20240502 02:02:11.468909  1492 raft_consensus.cc:348] T b80d547302ec453aaf80fa6b64eebc13 P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:11.469102  1491 ts_tablet_manager.cc:1398] T 2a1ce57d725948239c211b4200d5f037 P d0e4135c07ac4f2389ffe4e7379df71f: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 02:02:11.469419  1491 tablet_bootstrap.cc:492] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f: Bootstrap starting.
I20240502 02:02:11.469671  1492 raft_consensus.cc:374] T b80d547302ec453aaf80fa6b64eebc13 P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:02:11.469847  1492 raft_consensus.cc:729] T b80d547302ec453aaf80fa6b64eebc13 P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d37e50431f30463c872c5d819ddf2c25, State: Initialized, Role: FOLLOWER
I20240502 02:02:11.470223  1492 consensus_queue.cc:260] T b80d547302ec453aaf80fa6b64eebc13 P d37e50431f30463c872c5d819ddf2c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:11.471143  1491 tablet_bootstrap.cc:654] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f: Neither blocks nor log segments found. Creating new log.
I20240502 02:02:11.472090  1492 ts_tablet_manager.cc:1398] T b80d547302ec453aaf80fa6b64eebc13 P d37e50431f30463c872c5d819ddf2c25: Time spent starting tablet: real 0.005s	user 0.001s	sys 0.001s
I20240502 02:02:11.472467  1492 tablet_bootstrap.cc:492] T 2a1ce57d725948239c211b4200d5f037 P d37e50431f30463c872c5d819ddf2c25: Bootstrap starting.
I20240502 02:02:11.476188  1492 tablet_bootstrap.cc:654] T 2a1ce57d725948239c211b4200d5f037 P d37e50431f30463c872c5d819ddf2c25: Neither blocks nor log segments found. Creating new log.
I20240502 02:02:11.477999  1493 tablet_bootstrap.cc:492] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494: No bootstrap required, opened a new log
I20240502 02:02:11.478215  1493 ts_tablet_manager.cc:1367] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494: Time spent bootstrapping tablet: real 0.012s	user 0.003s	sys 0.006s
I20240502 02:02:11.479508  1491 tablet_bootstrap.cc:492] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f: No bootstrap required, opened a new log
I20240502 02:02:11.479462  1493 raft_consensus.cc:348] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:11.479700  1493 raft_consensus.cc:374] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:02:11.479714  1491 ts_tablet_manager.cc:1367] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f: Time spent bootstrapping tablet: real 0.010s	user 0.008s	sys 0.000s
I20240502 02:02:11.479797  1493 raft_consensus.cc:729] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b9e878e72f64482bdd89f28ac427494, State: Initialized, Role: FOLLOWER
I20240502 02:02:11.480388  1493 consensus_queue.cc:260] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:11.480537  1491 raft_consensus.cc:348] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:11.480798  1491 raft_consensus.cc:374] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:02:11.480890  1491 raft_consensus.cc:729] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0e4135c07ac4f2389ffe4e7379df71f, State: Initialized, Role: FOLLOWER
I20240502 02:02:11.481045  1493 ts_tablet_manager.cc:1398] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 02:02:11.481210  1491 consensus_queue.cc:260] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:11.481387  1493 tablet_bootstrap.cc:492] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494: Bootstrap starting.
I20240502 02:02:11.481954  1491 ts_tablet_manager.cc:1398] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 02:02:11.483322  1493 tablet_bootstrap.cc:654] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494: Neither blocks nor log segments found. Creating new log.
I20240502 02:02:11.485320  1492 tablet_bootstrap.cc:492] T 2a1ce57d725948239c211b4200d5f037 P d37e50431f30463c872c5d819ddf2c25: No bootstrap required, opened a new log
I20240502 02:02:11.485615  1492 ts_tablet_manager.cc:1367] T 2a1ce57d725948239c211b4200d5f037 P d37e50431f30463c872c5d819ddf2c25: Time spent bootstrapping tablet: real 0.013s	user 0.007s	sys 0.003s
I20240502 02:02:11.486541  1492 raft_consensus.cc:348] T 2a1ce57d725948239c211b4200d5f037 P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } }
I20240502 02:02:11.486960  1492 raft_consensus.cc:374] T 2a1ce57d725948239c211b4200d5f037 P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:02:11.486757  1289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e7759be006b4a8186aca434c48949b2" candidate_uuid: "8b9e878e72f64482bdd89f28ac427494" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d37e50431f30463c872c5d819ddf2c25" is_pre_election: true
I20240502 02:02:11.487445  1289 raft_consensus.cc:2454] T 9e7759be006b4a8186aca434c48949b2 P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b9e878e72f64482bdd89f28ac427494 in term 0.
I20240502 02:02:11.488255  1086 leader_election.cc:302] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b9e878e72f64482bdd89f28ac427494, d37e50431f30463c872c5d819ddf2c25; no voters: 
I20240502 02:02:11.488441  1020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e7759be006b4a8186aca434c48949b2" candidate_uuid: "8b9e878e72f64482bdd89f28ac427494" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" is_pre_election: true
I20240502 02:02:11.488914  1500 raft_consensus.cc:2781] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 02:02:11.488998  1020 raft_consensus.cc:2454] T 9e7759be006b4a8186aca434c48949b2 P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b9e878e72f64482bdd89f28ac427494 in term 0.
I20240502 02:02:11.489068  1500 raft_consensus.cc:482] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 02:02:11.489190  1500 raft_consensus.cc:3037] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:02:11.489470  1492 raft_consensus.cc:729] T 2a1ce57d725948239c211b4200d5f037 P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d37e50431f30463c872c5d819ddf2c25, State: Initialized, Role: FOLLOWER
I20240502 02:02:11.489887  1492 consensus_queue.cc:260] T 2a1ce57d725948239c211b4200d5f037 P d37e50431f30463c872c5d819ddf2c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } }
I20240502 02:02:11.491053  1492 ts_tablet_manager.cc:1398] T 2a1ce57d725948239c211b4200d5f037 P d37e50431f30463c872c5d819ddf2c25: Time spent starting tablet: real 0.005s	user 0.001s	sys 0.001s
I20240502 02:02:11.491740  1493 tablet_bootstrap.cc:492] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494: No bootstrap required, opened a new log
I20240502 02:02:11.491971  1493 ts_tablet_manager.cc:1367] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494: Time spent bootstrapping tablet: real 0.011s	user 0.009s	sys 0.001s
I20240502 02:02:11.492849  1493 raft_consensus.cc:348] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } }
I20240502 02:02:11.493060  1493 raft_consensus.cc:374] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:02:11.493192  1493 raft_consensus.cc:729] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b9e878e72f64482bdd89f28ac427494, State: Initialized, Role: FOLLOWER
I20240502 02:02:11.493467  1493 consensus_queue.cc:260] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } }
I20240502 02:02:11.493949  1500 raft_consensus.cc:504] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:11.494186  1493 ts_tablet_manager.cc:1398] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 02:02:11.494706  1500 leader_election.cc:288] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [CANDIDATE]: Term 1 election: Requested vote from peers d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129:33963), d37e50431f30463c872c5d819ddf2c25 (127.25.109.131:37799)
I20240502 02:02:11.495177  1020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e7759be006b4a8186aca434c48949b2" candidate_uuid: "8b9e878e72f64482bdd89f28ac427494" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e4135c07ac4f2389ffe4e7379df71f"
I20240502 02:02:11.495422  1289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e7759be006b4a8186aca434c48949b2" candidate_uuid: "8b9e878e72f64482bdd89f28ac427494" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d37e50431f30463c872c5d819ddf2c25"
I20240502 02:02:11.495460  1020 raft_consensus.cc:3037] T 9e7759be006b4a8186aca434c48949b2 P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:02:11.495626  1289 raft_consensus.cc:3037] T 9e7759be006b4a8186aca434c48949b2 P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:02:11.497929  1289 raft_consensus.cc:2454] T 9e7759be006b4a8186aca434c48949b2 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b9e878e72f64482bdd89f28ac427494 in term 1.
I20240502 02:02:11.498049  1020 raft_consensus.cc:2454] T 9e7759be006b4a8186aca434c48949b2 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b9e878e72f64482bdd89f28ac427494 in term 1.
I20240502 02:02:11.498435  1086 leader_election.cc:302] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b9e878e72f64482bdd89f28ac427494, d37e50431f30463c872c5d819ddf2c25; no voters: 
I20240502 02:02:11.498772  1500 raft_consensus.cc:2781] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:02:11.498982  1500 raft_consensus.cc:686] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [term 1 LEADER]: Becoming Leader. State: Replica: 8b9e878e72f64482bdd89f28ac427494, State: Running, Role: LEADER
I20240502 02:02:11.499547  1500 consensus_queue.cc:237] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:11.504458   875 catalog_manager.cc:5484] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130). New cstate: current_term: 1 leader_uuid: "8b9e878e72f64482bdd89f28ac427494" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } health_report { overall_health: UNKNOWN } } }
W20240502 02:02:11.600663  1335 tablet.cc:2369] T 2a1ce57d725948239c211b4200d5f037 P d37e50431f30463c872c5d819ddf2c25: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 02:02:11.631568  1500 raft_consensus.cc:482] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 02:02:11.631803  1500 raft_consensus.cc:504] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:11.632755  1500 leader_election.cc:288] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129:33963), d37e50431f30463c872c5d819ddf2c25 (127.25.109.131:37799)
I20240502 02:02:11.633180  1020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b80d547302ec453aaf80fa6b64eebc13" candidate_uuid: "8b9e878e72f64482bdd89f28ac427494" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" is_pre_election: true
I20240502 02:02:11.633288  1289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b80d547302ec453aaf80fa6b64eebc13" candidate_uuid: "8b9e878e72f64482bdd89f28ac427494" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d37e50431f30463c872c5d819ddf2c25" is_pre_election: true
I20240502 02:02:11.633451  1020 raft_consensus.cc:2454] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b9e878e72f64482bdd89f28ac427494 in term 0.
I20240502 02:02:11.633590  1289 raft_consensus.cc:2454] T b80d547302ec453aaf80fa6b64eebc13 P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b9e878e72f64482bdd89f28ac427494 in term 0.
I20240502 02:02:11.633931  1087 leader_election.cc:302] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b9e878e72f64482bdd89f28ac427494, d0e4135c07ac4f2389ffe4e7379df71f; no voters: 
I20240502 02:02:11.634207  1500 raft_consensus.cc:2781] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 02:02:11.634315  1500 raft_consensus.cc:482] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 02:02:11.634382  1500 raft_consensus.cc:3037] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:02:11.636389  1500 raft_consensus.cc:504] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:11.636998  1500 leader_election.cc:288] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [CANDIDATE]: Term 1 election: Requested vote from peers d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129:33963), d37e50431f30463c872c5d819ddf2c25 (127.25.109.131:37799)
I20240502 02:02:11.637614  1020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b80d547302ec453aaf80fa6b64eebc13" candidate_uuid: "8b9e878e72f64482bdd89f28ac427494" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e4135c07ac4f2389ffe4e7379df71f"
I20240502 02:02:11.637614  1289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b80d547302ec453aaf80fa6b64eebc13" candidate_uuid: "8b9e878e72f64482bdd89f28ac427494" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d37e50431f30463c872c5d819ddf2c25"
I20240502 02:02:11.637840  1020 raft_consensus.cc:3037] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:02:11.637866  1289 raft_consensus.cc:3037] T b80d547302ec453aaf80fa6b64eebc13 P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:02:11.639928  1289 raft_consensus.cc:2454] T b80d547302ec453aaf80fa6b64eebc13 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b9e878e72f64482bdd89f28ac427494 in term 1.
I20240502 02:02:11.639916  1020 raft_consensus.cc:2454] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b9e878e72f64482bdd89f28ac427494 in term 1.
I20240502 02:02:11.640401  1087 leader_election.cc:302] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b9e878e72f64482bdd89f28ac427494, d0e4135c07ac4f2389ffe4e7379df71f; no voters: 
I20240502 02:02:11.640723  1500 raft_consensus.cc:2781] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:02:11.640863  1500 raft_consensus.cc:686] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [term 1 LEADER]: Becoming Leader. State: Replica: 8b9e878e72f64482bdd89f28ac427494, State: Running, Role: LEADER
I20240502 02:02:11.641155  1500 consensus_queue.cc:237] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:11.643946   875 catalog_manager.cc:5484] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130). New cstate: current_term: 1 leader_uuid: "8b9e878e72f64482bdd89f28ac427494" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } health_report { overall_health: UNKNOWN } } }
W20240502 02:02:11.650743  1202 tablet.cc:2369] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 02:02:11.736428  1500 raft_consensus.cc:482] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 02:02:11.736691  1500 raft_consensus.cc:504] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } }
I20240502 02:02:11.737344  1500 leader_election.cc:288] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d37e50431f30463c872c5d819ddf2c25 (127.25.109.131:37799), d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129:33963)
I20240502 02:02:11.738047  1020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a1ce57d725948239c211b4200d5f037" candidate_uuid: "8b9e878e72f64482bdd89f28ac427494" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" is_pre_election: true
I20240502 02:02:11.738046  1289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a1ce57d725948239c211b4200d5f037" candidate_uuid: "8b9e878e72f64482bdd89f28ac427494" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d37e50431f30463c872c5d819ddf2c25" is_pre_election: true
I20240502 02:02:11.738310  1289 raft_consensus.cc:2454] T 2a1ce57d725948239c211b4200d5f037 P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b9e878e72f64482bdd89f28ac427494 in term 0.
I20240502 02:02:11.738310  1020 raft_consensus.cc:2454] T 2a1ce57d725948239c211b4200d5f037 P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b9e878e72f64482bdd89f28ac427494 in term 0.
I20240502 02:02:11.738796  1087 leader_election.cc:302] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8b9e878e72f64482bdd89f28ac427494, d0e4135c07ac4f2389ffe4e7379df71f, d37e50431f30463c872c5d819ddf2c25; no voters: 
I20240502 02:02:11.739138  1500 raft_consensus.cc:2781] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 02:02:11.739231  1500 raft_consensus.cc:482] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 02:02:11.739326  1500 raft_consensus.cc:3037] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:02:11.741261  1500 raft_consensus.cc:504] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } }
I20240502 02:02:11.741899  1500 leader_election.cc:288] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494 [CANDIDATE]: Term 1 election: Requested vote from peers d37e50431f30463c872c5d819ddf2c25 (127.25.109.131:37799), d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129:33963)
I20240502 02:02:11.742460  1289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a1ce57d725948239c211b4200d5f037" candidate_uuid: "8b9e878e72f64482bdd89f28ac427494" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d37e50431f30463c872c5d819ddf2c25"
I20240502 02:02:11.742529  1020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a1ce57d725948239c211b4200d5f037" candidate_uuid: "8b9e878e72f64482bdd89f28ac427494" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e4135c07ac4f2389ffe4e7379df71f"
I20240502 02:02:11.742717  1289 raft_consensus.cc:3037] T 2a1ce57d725948239c211b4200d5f037 P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:02:11.742727  1020 raft_consensus.cc:3037] T 2a1ce57d725948239c211b4200d5f037 P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:02:11.745610  1289 raft_consensus.cc:2454] T 2a1ce57d725948239c211b4200d5f037 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b9e878e72f64482bdd89f28ac427494 in term 1.
I20240502 02:02:11.745610  1020 raft_consensus.cc:2454] T 2a1ce57d725948239c211b4200d5f037 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b9e878e72f64482bdd89f28ac427494 in term 1.
I20240502 02:02:11.746155  1087 leader_election.cc:302] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b9e878e72f64482bdd89f28ac427494, d0e4135c07ac4f2389ffe4e7379df71f; no voters: 
I20240502 02:02:11.746474  1500 raft_consensus.cc:2781] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:02:11.746652  1500 raft_consensus.cc:686] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494 [term 1 LEADER]: Becoming Leader. State: Replica: 8b9e878e72f64482bdd89f28ac427494, State: Running, Role: LEADER
I20240502 02:02:11.746958  1500 consensus_queue.cc:237] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } }
I20240502 02:02:11.749894   875 catalog_manager.cc:5484] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130). New cstate: current_term: 1 leader_uuid: "8b9e878e72f64482bdd89f28ac427494" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } health_report { overall_health: UNKNOWN } } }
I20240502 02:02:11.927917  1500 consensus_queue.cc:1035] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:02:11.945494  1497 consensus_queue.cc:1035] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:02:12.153355  1497 consensus_queue.cc:1035] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 02:02:12.155001  1512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:02:12.155447  1512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 02:02:12.158002  1497 consensus_queue.cc:1035] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 02:02:12.165655  1512 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 02:02:12.199497  1500 consensus_queue.cc:1035] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240502 02:02:12.205904  1497 consensus_queue.cc:1035] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 217.784 ms
  time per row: 0.108892 ms
I20240502 02:02:12.538267   872 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42822:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240502 02:02:12.539095   872 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 02:02:12.547571   997 tablet_service.cc:1467] Processing CreateTablet for tablet f55670daadbc4a7f88971471629c793f (DEFAULT_TABLE table=rebalance_test_table_1 [id=8991230805234f99bb3820fc5a537d77]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 02:02:12.547785  1134 tablet_service.cc:1467] Processing CreateTablet for tablet f55670daadbc4a7f88971471629c793f (DEFAULT_TABLE table=rebalance_test_table_1 [id=8991230805234f99bb3820fc5a537d77]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 02:02:12.548142   997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f55670daadbc4a7f88971471629c793f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:12.548250  1134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f55670daadbc4a7f88971471629c793f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:12.548452  1267 tablet_service.cc:1467] Processing CreateTablet for tablet f55670daadbc4a7f88971471629c793f (DEFAULT_TABLE table=rebalance_test_table_1 [id=8991230805234f99bb3820fc5a537d77]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 02:02:12.548892  1267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f55670daadbc4a7f88971471629c793f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:12.550253  1268 tablet_service.cc:1467] Processing CreateTablet for tablet b1005c925b954ca39a9ee6f2677b3a1f (DEFAULT_TABLE table=rebalance_test_table_1 [id=8991230805234f99bb3820fc5a537d77]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 02:02:12.550634  1268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1005c925b954ca39a9ee6f2677b3a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:12.552287  1269 tablet_service.cc:1467] Processing CreateTablet for tablet 8573fb0d09af421ebbaeb98759e85f4d (DEFAULT_TABLE table=rebalance_test_table_1 [id=8991230805234f99bb3820fc5a537d77]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 02:02:12.552585   998 tablet_service.cc:1467] Processing CreateTablet for tablet 8573fb0d09af421ebbaeb98759e85f4d (DEFAULT_TABLE table=rebalance_test_table_1 [id=8991230805234f99bb3820fc5a537d77]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 02:02:12.552764  1269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8573fb0d09af421ebbaeb98759e85f4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:12.552965   998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8573fb0d09af421ebbaeb98759e85f4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:12.556254   999 tablet_service.cc:1467] Processing CreateTablet for tablet b1005c925b954ca39a9ee6f2677b3a1f (DEFAULT_TABLE table=rebalance_test_table_1 [id=8991230805234f99bb3820fc5a537d77]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 02:02:12.556617   999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1005c925b954ca39a9ee6f2677b3a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:12.556644  1135 tablet_service.cc:1467] Processing CreateTablet for tablet 8573fb0d09af421ebbaeb98759e85f4d (DEFAULT_TABLE table=rebalance_test_table_1 [id=8991230805234f99bb3820fc5a537d77]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 02:02:12.557018  1135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8573fb0d09af421ebbaeb98759e85f4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:12.560258  1136 tablet_service.cc:1467] Processing CreateTablet for tablet b1005c925b954ca39a9ee6f2677b3a1f (DEFAULT_TABLE table=rebalance_test_table_1 [id=8991230805234f99bb3820fc5a537d77]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 02:02:12.560645  1136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1005c925b954ca39a9ee6f2677b3a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:12.565243  1576 tablet_bootstrap.cc:492] T f55670daadbc4a7f88971471629c793f P d37e50431f30463c872c5d819ddf2c25: Bootstrap starting.
I20240502 02:02:12.565996  1577 tablet_bootstrap.cc:492] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f: Bootstrap starting.
I20240502 02:02:12.567396  1576 tablet_bootstrap.cc:654] T f55670daadbc4a7f88971471629c793f P d37e50431f30463c872c5d819ddf2c25: Neither blocks nor log segments found. Creating new log.
I20240502 02:02:12.567783  1577 tablet_bootstrap.cc:654] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f: Neither blocks nor log segments found. Creating new log.
I20240502 02:02:12.569594  1576 tablet_bootstrap.cc:492] T f55670daadbc4a7f88971471629c793f P d37e50431f30463c872c5d819ddf2c25: No bootstrap required, opened a new log
I20240502 02:02:12.569833  1576 ts_tablet_manager.cc:1367] T f55670daadbc4a7f88971471629c793f P d37e50431f30463c872c5d819ddf2c25: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240502 02:02:12.570719  1576 raft_consensus.cc:348] T f55670daadbc4a7f88971471629c793f P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:12.571072  1576 raft_consensus.cc:374] T f55670daadbc4a7f88971471629c793f P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:02:12.571244  1576 raft_consensus.cc:729] T f55670daadbc4a7f88971471629c793f P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d37e50431f30463c872c5d819ddf2c25, State: Initialized, Role: FOLLOWER
I20240502 02:02:12.571627  1576 consensus_queue.cc:260] T f55670daadbc4a7f88971471629c793f P d37e50431f30463c872c5d819ddf2c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:12.572129  1580 tablet_bootstrap.cc:492] T b1005c925b954ca39a9ee6f2677b3a1f P 8b9e878e72f64482bdd89f28ac427494: Bootstrap starting.
I20240502 02:02:12.573477  1576 ts_tablet_manager.cc:1398] T f55670daadbc4a7f88971471629c793f P d37e50431f30463c872c5d819ddf2c25: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 02:02:12.573827  1576 tablet_bootstrap.cc:492] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25: Bootstrap starting.
I20240502 02:02:12.576581  1580 tablet_bootstrap.cc:654] T b1005c925b954ca39a9ee6f2677b3a1f P 8b9e878e72f64482bdd89f28ac427494: Neither blocks nor log segments found. Creating new log.
I20240502 02:02:12.576606  1576 tablet_bootstrap.cc:654] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25: Neither blocks nor log segments found. Creating new log.
I20240502 02:02:12.579094  1577 tablet_bootstrap.cc:492] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f: No bootstrap required, opened a new log
I20240502 02:02:12.579275  1577 ts_tablet_manager.cc:1367] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f: Time spent bootstrapping tablet: real 0.013s	user 0.008s	sys 0.004s
I20240502 02:02:12.579295  1580 tablet_bootstrap.cc:492] T b1005c925b954ca39a9ee6f2677b3a1f P 8b9e878e72f64482bdd89f28ac427494: No bootstrap required, opened a new log
I20240502 02:02:12.579314  1576 tablet_bootstrap.cc:492] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25: No bootstrap required, opened a new log
I20240502 02:02:12.579442  1580 ts_tablet_manager.cc:1367] T b1005c925b954ca39a9ee6f2677b3a1f P 8b9e878e72f64482bdd89f28ac427494: Time spent bootstrapping tablet: real 0.007s	user 0.002s	sys 0.001s
I20240502 02:02:12.579615  1576 ts_tablet_manager.cc:1367] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25: Time spent bootstrapping tablet: real 0.006s	user 0.000s	sys 0.003s
I20240502 02:02:12.580116  1577 raft_consensus.cc:348] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:12.580300  1577 raft_consensus.cc:374] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:02:12.580391  1577 raft_consensus.cc:729] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0e4135c07ac4f2389ffe4e7379df71f, State: Initialized, Role: FOLLOWER
I20240502 02:02:12.580291  1580 raft_consensus.cc:348] T b1005c925b954ca39a9ee6f2677b3a1f P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:12.580510  1580 raft_consensus.cc:374] T b1005c925b954ca39a9ee6f2677b3a1f P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:02:12.580399  1576 raft_consensus.cc:348] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:12.580600  1580 raft_consensus.cc:729] T b1005c925b954ca39a9ee6f2677b3a1f P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b9e878e72f64482bdd89f28ac427494, State: Initialized, Role: FOLLOWER
I20240502 02:02:12.580617  1576 raft_consensus.cc:374] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:02:12.580689  1576 raft_consensus.cc:729] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d37e50431f30463c872c5d819ddf2c25, State: Initialized, Role: FOLLOWER
I20240502 02:02:12.580646  1577 consensus_queue.cc:260] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:12.580840  1580 consensus_queue.cc:260] T b1005c925b954ca39a9ee6f2677b3a1f P 8b9e878e72f64482bdd89f28ac427494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:12.581279  1577 ts_tablet_manager.cc:1398] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 02:02:12.581562  1577 tablet_bootstrap.cc:492] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f: Bootstrap starting.
I20240502 02:02:12.580948  1576 consensus_queue.cc:260] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:12.582038  1580 ts_tablet_manager.cc:1398] T b1005c925b954ca39a9ee6f2677b3a1f P 8b9e878e72f64482bdd89f28ac427494: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 02:02:12.582324  1580 tablet_bootstrap.cc:492] T f55670daadbc4a7f88971471629c793f P 8b9e878e72f64482bdd89f28ac427494: Bootstrap starting.
I20240502 02:02:12.583173  1576 ts_tablet_manager.cc:1398] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240502 02:02:12.583214  1577 tablet_bootstrap.cc:654] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f: Neither blocks nor log segments found. Creating new log.
I20240502 02:02:12.583451  1576 tablet_bootstrap.cc:492] T 8573fb0d09af421ebbaeb98759e85f4d P d37e50431f30463c872c5d819ddf2c25: Bootstrap starting.
I20240502 02:02:12.583884  1580 tablet_bootstrap.cc:654] T f55670daadbc4a7f88971471629c793f P 8b9e878e72f64482bdd89f28ac427494: Neither blocks nor log segments found. Creating new log.
I20240502 02:02:12.585050  1576 tablet_bootstrap.cc:654] T 8573fb0d09af421ebbaeb98759e85f4d P d37e50431f30463c872c5d819ddf2c25: Neither blocks nor log segments found. Creating new log.
I20240502 02:02:12.589067  1580 tablet_bootstrap.cc:492] T f55670daadbc4a7f88971471629c793f P 8b9e878e72f64482bdd89f28ac427494: No bootstrap required, opened a new log
I20240502 02:02:12.589283  1580 ts_tablet_manager.cc:1367] T f55670daadbc4a7f88971471629c793f P 8b9e878e72f64482bdd89f28ac427494: Time spent bootstrapping tablet: real 0.007s	user 0.002s	sys 0.004s
I20240502 02:02:12.590133  1580 raft_consensus.cc:348] T f55670daadbc4a7f88971471629c793f P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:12.590386  1580 raft_consensus.cc:374] T f55670daadbc4a7f88971471629c793f P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:02:12.590463  1580 raft_consensus.cc:729] T f55670daadbc4a7f88971471629c793f P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b9e878e72f64482bdd89f28ac427494, State: Initialized, Role: FOLLOWER
I20240502 02:02:12.590776  1580 consensus_queue.cc:260] T f55670daadbc4a7f88971471629c793f P 8b9e878e72f64482bdd89f28ac427494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:12.591413  1580 ts_tablet_manager.cc:1398] T f55670daadbc4a7f88971471629c793f P 8b9e878e72f64482bdd89f28ac427494: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240502 02:02:12.591709  1580 tablet_bootstrap.cc:492] T 8573fb0d09af421ebbaeb98759e85f4d P 8b9e878e72f64482bdd89f28ac427494: Bootstrap starting.
I20240502 02:02:12.593263  1576 tablet_bootstrap.cc:492] T 8573fb0d09af421ebbaeb98759e85f4d P d37e50431f30463c872c5d819ddf2c25: No bootstrap required, opened a new log
I20240502 02:02:12.593364  1580 tablet_bootstrap.cc:654] T 8573fb0d09af421ebbaeb98759e85f4d P 8b9e878e72f64482bdd89f28ac427494: Neither blocks nor log segments found. Creating new log.
I20240502 02:02:12.593441  1576 ts_tablet_manager.cc:1367] T 8573fb0d09af421ebbaeb98759e85f4d P d37e50431f30463c872c5d819ddf2c25: Time spent bootstrapping tablet: real 0.010s	user 0.009s	sys 0.000s
I20240502 02:02:12.594326  1576 raft_consensus.cc:348] T 8573fb0d09af421ebbaeb98759e85f4d P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:12.594545  1576 raft_consensus.cc:374] T 8573fb0d09af421ebbaeb98759e85f4d P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:02:12.594655  1576 raft_consensus.cc:729] T 8573fb0d09af421ebbaeb98759e85f4d P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d37e50431f30463c872c5d819ddf2c25, State: Initialized, Role: FOLLOWER
I20240502 02:02:12.594893  1576 consensus_queue.cc:260] T 8573fb0d09af421ebbaeb98759e85f4d P d37e50431f30463c872c5d819ddf2c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:12.595635  1577 tablet_bootstrap.cc:492] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f: No bootstrap required, opened a new log
I20240502 02:02:12.595822  1577 ts_tablet_manager.cc:1367] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f: Time spent bootstrapping tablet: real 0.014s	user 0.005s	sys 0.008s
I20240502 02:02:12.596213  1576 ts_tablet_manager.cc:1398] T 8573fb0d09af421ebbaeb98759e85f4d P d37e50431f30463c872c5d819ddf2c25: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 02:02:12.596673  1580 tablet_bootstrap.cc:492] T 8573fb0d09af421ebbaeb98759e85f4d P 8b9e878e72f64482bdd89f28ac427494: No bootstrap required, opened a new log
I20240502 02:02:12.596807  1580 ts_tablet_manager.cc:1367] T 8573fb0d09af421ebbaeb98759e85f4d P 8b9e878e72f64482bdd89f28ac427494: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240502 02:02:12.596715  1577 raft_consensus.cc:348] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:12.596951  1577 raft_consensus.cc:374] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:02:12.597043  1577 raft_consensus.cc:729] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0e4135c07ac4f2389ffe4e7379df71f, State: Initialized, Role: FOLLOWER
I20240502 02:02:12.597329  1577 consensus_queue.cc:260] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:12.597445  1580 raft_consensus.cc:348] T 8573fb0d09af421ebbaeb98759e85f4d P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:12.597623  1580 raft_consensus.cc:374] T 8573fb0d09af421ebbaeb98759e85f4d P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:02:12.597713  1580 raft_consensus.cc:729] T 8573fb0d09af421ebbaeb98759e85f4d P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b9e878e72f64482bdd89f28ac427494, State: Initialized, Role: FOLLOWER
I20240502 02:02:12.597887  1577 ts_tablet_manager.cc:1398] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 02:02:12.597916  1580 consensus_queue.cc:260] T 8573fb0d09af421ebbaeb98759e85f4d P 8b9e878e72f64482bdd89f28ac427494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:12.598219  1577 tablet_bootstrap.cc:492] T 8573fb0d09af421ebbaeb98759e85f4d P d0e4135c07ac4f2389ffe4e7379df71f: Bootstrap starting.
I20240502 02:02:12.598419  1580 ts_tablet_manager.cc:1398] T 8573fb0d09af421ebbaeb98759e85f4d P 8b9e878e72f64482bdd89f28ac427494: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 02:02:12.599889  1577 tablet_bootstrap.cc:654] T 8573fb0d09af421ebbaeb98759e85f4d P d0e4135c07ac4f2389ffe4e7379df71f: Neither blocks nor log segments found. Creating new log.
I20240502 02:02:12.604264  1577 tablet_bootstrap.cc:492] T 8573fb0d09af421ebbaeb98759e85f4d P d0e4135c07ac4f2389ffe4e7379df71f: No bootstrap required, opened a new log
I20240502 02:02:12.604398  1577 ts_tablet_manager.cc:1367] T 8573fb0d09af421ebbaeb98759e85f4d P d0e4135c07ac4f2389ffe4e7379df71f: Time spent bootstrapping tablet: real 0.006s	user 0.001s	sys 0.004s
I20240502 02:02:12.604990  1577 raft_consensus.cc:348] T 8573fb0d09af421ebbaeb98759e85f4d P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:12.605161  1577 raft_consensus.cc:374] T 8573fb0d09af421ebbaeb98759e85f4d P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:02:12.605250  1577 raft_consensus.cc:729] T 8573fb0d09af421ebbaeb98759e85f4d P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0e4135c07ac4f2389ffe4e7379df71f, State: Initialized, Role: FOLLOWER
I20240502 02:02:12.605450  1577 consensus_queue.cc:260] T 8573fb0d09af421ebbaeb98759e85f4d P d0e4135c07ac4f2389ffe4e7379df71f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:12.605937  1577 ts_tablet_manager.cc:1398] T 8573fb0d09af421ebbaeb98759e85f4d P d0e4135c07ac4f2389ffe4e7379df71f: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 02:02:12.614423  1499 raft_consensus.cc:482] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 02:02:12.614625  1499 raft_consensus.cc:504] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:12.616235  1499 leader_election.cc:288] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129:33963), 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130:43631)
I20240502 02:02:12.623404  1156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1005c925b954ca39a9ee6f2677b3a1f" candidate_uuid: "d37e50431f30463c872c5d819ddf2c25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b9e878e72f64482bdd89f28ac427494" is_pre_election: true
I20240502 02:02:12.623692  1019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1005c925b954ca39a9ee6f2677b3a1f" candidate_uuid: "d37e50431f30463c872c5d819ddf2c25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" is_pre_election: true
I20240502 02:02:12.623886  1156 raft_consensus.cc:2454] T b1005c925b954ca39a9ee6f2677b3a1f P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d37e50431f30463c872c5d819ddf2c25 in term 0.
I20240502 02:02:12.623942  1019 raft_consensus.cc:2454] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d37e50431f30463c872c5d819ddf2c25 in term 0.
I20240502 02:02:12.624470  1221 leader_election.cc:302] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0e4135c07ac4f2389ffe4e7379df71f, d37e50431f30463c872c5d819ddf2c25; no voters: 
I20240502 02:02:12.624651  1498 raft_consensus.cc:482] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 02:02:12.624903  1499 raft_consensus.cc:2781] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 02:02:12.624809  1498 raft_consensus.cc:504] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:12.625028  1499 raft_consensus.cc:482] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 02:02:12.625123  1499 raft_consensus.cc:3037] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:02:12.626399  1498 leader_election.cc:288] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130:43631), d37e50431f30463c872c5d819ddf2c25 (127.25.109.131:37799)
I20240502 02:02:12.627563  1499 raft_consensus.cc:504] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:12.629133  1156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1005c925b954ca39a9ee6f2677b3a1f" candidate_uuid: "d37e50431f30463c872c5d819ddf2c25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b9e878e72f64482bdd89f28ac427494"
I20240502 02:02:12.629385  1156 raft_consensus.cc:3037] T b1005c925b954ca39a9ee6f2677b3a1f P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:02:12.632668  1499 leader_election.cc:288] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [CANDIDATE]: Term 1 election: Requested vote from peers d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129:33963), 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130:43631)
I20240502 02:02:12.633034  1156 raft_consensus.cc:2454] T b1005c925b954ca39a9ee6f2677b3a1f P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d37e50431f30463c872c5d819ddf2c25 in term 1.
I20240502 02:02:12.633673  1222 leader_election.cc:302] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b9e878e72f64482bdd89f28ac427494, d37e50431f30463c872c5d819ddf2c25; no voters: 
I20240502 02:02:12.634074  1499 raft_consensus.cc:2781] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:02:12.637754  1499 raft_consensus.cc:686] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [term 1 LEADER]: Becoming Leader. State: Replica: d37e50431f30463c872c5d819ddf2c25, State: Running, Role: LEADER
I20240502 02:02:12.638255  1499 consensus_queue.cc:237] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:12.642439   871 catalog_manager.cc:5484] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 reported cstate change: term changed from 0 to 1, leader changed from <none> to d37e50431f30463c872c5d819ddf2c25 (127.25.109.131). New cstate: current_term: 1 leader_uuid: "d37e50431f30463c872c5d819ddf2c25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } health_report { overall_health: HEALTHY } } }
I20240502 02:02:12.643308  1019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1005c925b954ca39a9ee6f2677b3a1f" candidate_uuid: "d37e50431f30463c872c5d819ddf2c25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e4135c07ac4f2389ffe4e7379df71f"
I20240502 02:02:12.643594  1019 raft_consensus.cc:3037] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:02:12.646360  1019 raft_consensus.cc:2454] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d37e50431f30463c872c5d819ddf2c25 in term 1.
I20240502 02:02:12.649593  1156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55670daadbc4a7f88971471629c793f" candidate_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b9e878e72f64482bdd89f28ac427494" is_pre_election: true
I20240502 02:02:12.649880  1156 raft_consensus.cc:2454] T f55670daadbc4a7f88971471629c793f P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0e4135c07ac4f2389ffe4e7379df71f in term 0.
I20240502 02:02:12.649883  1288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55670daadbc4a7f88971471629c793f" candidate_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d37e50431f30463c872c5d819ddf2c25" is_pre_election: true
I20240502 02:02:12.650117  1288 raft_consensus.cc:2454] T f55670daadbc4a7f88971471629c793f P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0e4135c07ac4f2389ffe4e7379df71f in term 0.
I20240502 02:02:12.650427   953 leader_election.cc:302] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b9e878e72f64482bdd89f28ac427494, d0e4135c07ac4f2389ffe4e7379df71f; no voters: 
I20240502 02:02:12.650842  1498 raft_consensus.cc:2781] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 02:02:12.650972  1498 raft_consensus.cc:482] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 02:02:12.651077  1498 raft_consensus.cc:3037] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:02:12.653223  1498 raft_consensus.cc:504] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:12.653856  1498 leader_election.cc:288] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [CANDIDATE]: Term 1 election: Requested vote from peers 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130:43631), d37e50431f30463c872c5d819ddf2c25 (127.25.109.131:37799)
I20240502 02:02:12.654227  1156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55670daadbc4a7f88971471629c793f" candidate_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b9e878e72f64482bdd89f28ac427494"
I20240502 02:02:12.654467  1156 raft_consensus.cc:3037] T f55670daadbc4a7f88971471629c793f P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:02:12.654453  1288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55670daadbc4a7f88971471629c793f" candidate_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d37e50431f30463c872c5d819ddf2c25"
I20240502 02:02:12.654659  1288 raft_consensus.cc:3037] T f55670daadbc4a7f88971471629c793f P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:02:12.657270  1156 raft_consensus.cc:2454] T f55670daadbc4a7f88971471629c793f P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0e4135c07ac4f2389ffe4e7379df71f in term 1.
I20240502 02:02:12.657322  1288 raft_consensus.cc:2454] T f55670daadbc4a7f88971471629c793f P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0e4135c07ac4f2389ffe4e7379df71f in term 1.
I20240502 02:02:12.657725   953 leader_election.cc:302] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b9e878e72f64482bdd89f28ac427494, d0e4135c07ac4f2389ffe4e7379df71f; no voters: 
I20240502 02:02:12.658031  1498 raft_consensus.cc:2781] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:02:12.658480  1498 raft_consensus.cc:686] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [term 1 LEADER]: Becoming Leader. State: Replica: d0e4135c07ac4f2389ffe4e7379df71f, State: Running, Role: LEADER
I20240502 02:02:12.658934  1498 consensus_queue.cc:237] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:12.662606   871 catalog_manager.cc:5484] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f reported cstate change: term changed from 0 to 1, leader changed from <none> to d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129). New cstate: current_term: 1 leader_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } health_report { overall_health: UNKNOWN } } }
I20240502 02:02:12.891845  1552 raft_consensus.cc:482] T 8573fb0d09af421ebbaeb98759e85f4d P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 02:02:12.892114  1552 raft_consensus.cc:504] T 8573fb0d09af421ebbaeb98759e85f4d P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:12.892815  1552 leader_election.cc:288] T 8573fb0d09af421ebbaeb98759e85f4d P 8b9e878e72f64482bdd89f28ac427494 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129:33963), d37e50431f30463c872c5d819ddf2c25 (127.25.109.131:37799)
I20240502 02:02:12.893369  1019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8573fb0d09af421ebbaeb98759e85f4d" candidate_uuid: "8b9e878e72f64482bdd89f28ac427494" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" is_pre_election: true
I20240502 02:02:12.893399  1288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8573fb0d09af421ebbaeb98759e85f4d" candidate_uuid: "8b9e878e72f64482bdd89f28ac427494" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d37e50431f30463c872c5d819ddf2c25" is_pre_election: true
I20240502 02:02:12.893620  1019 raft_consensus.cc:2454] T 8573fb0d09af421ebbaeb98759e85f4d P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b9e878e72f64482bdd89f28ac427494 in term 0.
I20240502 02:02:12.893647  1288 raft_consensus.cc:2454] T 8573fb0d09af421ebbaeb98759e85f4d P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b9e878e72f64482bdd89f28ac427494 in term 0.
I20240502 02:02:12.894065  1086 leader_election.cc:302] T 8573fb0d09af421ebbaeb98759e85f4d P 8b9e878e72f64482bdd89f28ac427494 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b9e878e72f64482bdd89f28ac427494, d37e50431f30463c872c5d819ddf2c25; no voters: 
I20240502 02:02:12.894381  1552 raft_consensus.cc:2781] T 8573fb0d09af421ebbaeb98759e85f4d P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 02:02:12.894505  1552 raft_consensus.cc:482] T 8573fb0d09af421ebbaeb98759e85f4d P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 02:02:12.894593  1552 raft_consensus.cc:3037] T 8573fb0d09af421ebbaeb98759e85f4d P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:02:12.896590  1552 raft_consensus.cc:504] T 8573fb0d09af421ebbaeb98759e85f4d P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:12.897150  1552 leader_election.cc:288] T 8573fb0d09af421ebbaeb98759e85f4d P 8b9e878e72f64482bdd89f28ac427494 [CANDIDATE]: Term 1 election: Requested vote from peers d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129:33963), d37e50431f30463c872c5d819ddf2c25 (127.25.109.131:37799)
I20240502 02:02:12.897675  1019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8573fb0d09af421ebbaeb98759e85f4d" candidate_uuid: "8b9e878e72f64482bdd89f28ac427494" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e4135c07ac4f2389ffe4e7379df71f"
I20240502 02:02:12.897675  1288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8573fb0d09af421ebbaeb98759e85f4d" candidate_uuid: "8b9e878e72f64482bdd89f28ac427494" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d37e50431f30463c872c5d819ddf2c25"
I20240502 02:02:12.897915  1019 raft_consensus.cc:3037] T 8573fb0d09af421ebbaeb98759e85f4d P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:02:12.897915  1288 raft_consensus.cc:3037] T 8573fb0d09af421ebbaeb98759e85f4d P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:02:12.900681  1288 raft_consensus.cc:2454] T 8573fb0d09af421ebbaeb98759e85f4d P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b9e878e72f64482bdd89f28ac427494 in term 1.
I20240502 02:02:12.900679  1019 raft_consensus.cc:2454] T 8573fb0d09af421ebbaeb98759e85f4d P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b9e878e72f64482bdd89f28ac427494 in term 1.
I20240502 02:02:12.901105  1086 leader_election.cc:302] T 8573fb0d09af421ebbaeb98759e85f4d P 8b9e878e72f64482bdd89f28ac427494 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b9e878e72f64482bdd89f28ac427494, d37e50431f30463c872c5d819ddf2c25; no voters: 
I20240502 02:02:12.901412  1552 raft_consensus.cc:2781] T 8573fb0d09af421ebbaeb98759e85f4d P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:02:12.901557  1552 raft_consensus.cc:686] T 8573fb0d09af421ebbaeb98759e85f4d P 8b9e878e72f64482bdd89f28ac427494 [term 1 LEADER]: Becoming Leader. State: Replica: 8b9e878e72f64482bdd89f28ac427494, State: Running, Role: LEADER
I20240502 02:02:12.901799  1552 consensus_queue.cc:237] T 8573fb0d09af421ebbaeb98759e85f4d P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:12.904508   871 catalog_manager.cc:5484] T 8573fb0d09af421ebbaeb98759e85f4d P 8b9e878e72f64482bdd89f28ac427494 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130). New cstate: current_term: 1 leader_uuid: "8b9e878e72f64482bdd89f28ac427494" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } health_report { overall_health: UNKNOWN } } }
I20240502 02:02:13.051445  1499 consensus_queue.cc:1035] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:02:13.057680  1499 consensus_queue.cc:1035] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:02:13.100732  1498 consensus_queue.cc:1035] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:02:13.105952  1498 consensus_queue.cc:1035] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 02:02:13.285923  1599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:02:13.286275  1599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:02:13.294037  1599 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 02:02:13.460233  1288 raft_consensus.cc:1261] T 8573fb0d09af421ebbaeb98759e85f4d P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Refusing update from remote peer 8b9e878e72f64482bdd89f28ac427494: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 02:02:13.461004  1552 consensus_queue.cc:1035] T 8573fb0d09af421ebbaeb98759e85f4d P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:02:13.466190  1552 consensus_queue.cc:1035] T 8573fb0d09af421ebbaeb98759e85f4d P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 247.836 ms
  time per row: 0.123918 ms
I20240502 02:02:13.692706   871 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42822:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240502 02:02:13.693468   871 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 02:02:13.702003  1135 tablet_service.cc:1467] Processing CreateTablet for tablet 11210087d9164d2c9c1303e659230192 (DEFAULT_TABLE table=rebalance_test_table_2 [id=57c9d0a8579e4e908683ee50f24d5aa1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 02:02:13.702253  1269 tablet_service.cc:1467] Processing CreateTablet for tablet 11210087d9164d2c9c1303e659230192 (DEFAULT_TABLE table=rebalance_test_table_2 [id=57c9d0a8579e4e908683ee50f24d5aa1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 02:02:13.702535  1135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11210087d9164d2c9c1303e659230192. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:13.702682  1269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11210087d9164d2c9c1303e659230192. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:13.702731  1267 tablet_service.cc:1467] Processing CreateTablet for tablet 884b0248d66f4575a7e8e3ead42c58c6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=57c9d0a8579e4e908683ee50f24d5aa1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 02:02:13.703161  1267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 884b0248d66f4575a7e8e3ead42c58c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:13.703223   999 tablet_service.cc:1467] Processing CreateTablet for tablet 11210087d9164d2c9c1303e659230192 (DEFAULT_TABLE table=rebalance_test_table_2 [id=57c9d0a8579e4e908683ee50f24d5aa1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 02:02:13.703625   999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11210087d9164d2c9c1303e659230192. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:13.708293   998 tablet_service.cc:1467] Processing CreateTablet for tablet 884b0248d66f4575a7e8e3ead42c58c6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=57c9d0a8579e4e908683ee50f24d5aa1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 02:02:13.708691   998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 884b0248d66f4575a7e8e3ead42c58c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:13.711330  1668 tablet_bootstrap.cc:492] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f: Bootstrap starting.
I20240502 02:02:13.712450  1268 tablet_service.cc:1467] Processing CreateTablet for tablet 53d5bc373eb14d44bf1dd07375f40f00 (DEFAULT_TABLE table=rebalance_test_table_2 [id=57c9d0a8579e4e908683ee50f24d5aa1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 02:02:13.712994  1268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53d5bc373eb14d44bf1dd07375f40f00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:13.713124  1668 tablet_bootstrap.cc:654] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f: Neither blocks nor log segments found. Creating new log.
I20240502 02:02:13.713382   997 tablet_service.cc:1467] Processing CreateTablet for tablet 53d5bc373eb14d44bf1dd07375f40f00 (DEFAULT_TABLE table=rebalance_test_table_2 [id=57c9d0a8579e4e908683ee50f24d5aa1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 02:02:13.713781   997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53d5bc373eb14d44bf1dd07375f40f00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:13.718222  1134 tablet_service.cc:1467] Processing CreateTablet for tablet 53d5bc373eb14d44bf1dd07375f40f00 (DEFAULT_TABLE table=rebalance_test_table_2 [id=57c9d0a8579e4e908683ee50f24d5aa1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 02:02:13.718254  1136 tablet_service.cc:1467] Processing CreateTablet for tablet 884b0248d66f4575a7e8e3ead42c58c6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=57c9d0a8579e4e908683ee50f24d5aa1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 02:02:13.718670  1134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53d5bc373eb14d44bf1dd07375f40f00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:13.718897  1136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 884b0248d66f4575a7e8e3ead42c58c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:13.729311  1670 tablet_bootstrap.cc:492] T 53d5bc373eb14d44bf1dd07375f40f00 P 8b9e878e72f64482bdd89f28ac427494: Bootstrap starting.
I20240502 02:02:13.729359  1667 tablet_bootstrap.cc:492] T 11210087d9164d2c9c1303e659230192 P d37e50431f30463c872c5d819ddf2c25: Bootstrap starting.
I20240502 02:02:13.730616  1668 tablet_bootstrap.cc:492] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f: No bootstrap required, opened a new log
I20240502 02:02:13.730769  1668 ts_tablet_manager.cc:1367] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f: Time spent bootstrapping tablet: real 0.020s	user 0.009s	sys 0.000s
I20240502 02:02:13.731077  1670 tablet_bootstrap.cc:654] T 53d5bc373eb14d44bf1dd07375f40f00 P 8b9e878e72f64482bdd89f28ac427494: Neither blocks nor log segments found. Creating new log.
I20240502 02:02:13.731087  1667 tablet_bootstrap.cc:654] T 11210087d9164d2c9c1303e659230192 P d37e50431f30463c872c5d819ddf2c25: Neither blocks nor log segments found. Creating new log.
I20240502 02:02:13.731459  1668 raft_consensus.cc:348] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:13.731657  1668 raft_consensus.cc:374] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:02:13.731739  1668 raft_consensus.cc:729] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0e4135c07ac4f2389ffe4e7379df71f, State: Initialized, Role: FOLLOWER
I20240502 02:02:13.732017  1668 consensus_queue.cc:260] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:13.733882  1668 ts_tablet_manager.cc:1398] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 02:02:13.734151  1668 tablet_bootstrap.cc:492] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f: Bootstrap starting.
I20240502 02:02:13.734203  1670 tablet_bootstrap.cc:492] T 53d5bc373eb14d44bf1dd07375f40f00 P 8b9e878e72f64482bdd89f28ac427494: No bootstrap required, opened a new log
I20240502 02:02:13.734376  1670 ts_tablet_manager.cc:1367] T 53d5bc373eb14d44bf1dd07375f40f00 P 8b9e878e72f64482bdd89f28ac427494: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240502 02:02:13.735179  1670 raft_consensus.cc:348] T 53d5bc373eb14d44bf1dd07375f40f00 P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:13.735389  1670 raft_consensus.cc:374] T 53d5bc373eb14d44bf1dd07375f40f00 P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:02:13.735494  1670 raft_consensus.cc:729] T 53d5bc373eb14d44bf1dd07375f40f00 P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b9e878e72f64482bdd89f28ac427494, State: Initialized, Role: FOLLOWER
I20240502 02:02:13.735733  1668 tablet_bootstrap.cc:654] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f: Neither blocks nor log segments found. Creating new log.
I20240502 02:02:13.735759  1670 consensus_queue.cc:260] T 53d5bc373eb14d44bf1dd07375f40f00 P 8b9e878e72f64482bdd89f28ac427494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:13.736089  1667 tablet_bootstrap.cc:492] T 11210087d9164d2c9c1303e659230192 P d37e50431f30463c872c5d819ddf2c25: No bootstrap required, opened a new log
I20240502 02:02:13.736279  1667 ts_tablet_manager.cc:1367] T 11210087d9164d2c9c1303e659230192 P d37e50431f30463c872c5d819ddf2c25: Time spent bootstrapping tablet: real 0.007s	user 0.005s	sys 0.000s
I20240502 02:02:13.736362  1670 ts_tablet_manager.cc:1398] T 53d5bc373eb14d44bf1dd07375f40f00 P 8b9e878e72f64482bdd89f28ac427494: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 02:02:13.737250  1670 tablet_bootstrap.cc:492] T 11210087d9164d2c9c1303e659230192 P 8b9e878e72f64482bdd89f28ac427494: Bootstrap starting.
I20240502 02:02:13.737277  1667 raft_consensus.cc:348] T 11210087d9164d2c9c1303e659230192 P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:13.737699  1667 raft_consensus.cc:374] T 11210087d9164d2c9c1303e659230192 P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:02:13.737884  1667 raft_consensus.cc:729] T 11210087d9164d2c9c1303e659230192 P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d37e50431f30463c872c5d819ddf2c25, State: Initialized, Role: FOLLOWER
I20240502 02:02:13.738196  1667 consensus_queue.cc:260] T 11210087d9164d2c9c1303e659230192 P d37e50431f30463c872c5d819ddf2c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:13.739444  1667 ts_tablet_manager.cc:1398] T 11210087d9164d2c9c1303e659230192 P d37e50431f30463c872c5d819ddf2c25: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 02:02:13.739751  1667 tablet_bootstrap.cc:492] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25: Bootstrap starting.
I20240502 02:02:13.740510  1668 tablet_bootstrap.cc:492] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f: No bootstrap required, opened a new log
I20240502 02:02:13.740746  1668 ts_tablet_manager.cc:1367] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f: Time spent bootstrapping tablet: real 0.007s	user 0.006s	sys 0.000s
I20240502 02:02:13.741158  1670 tablet_bootstrap.cc:654] T 11210087d9164d2c9c1303e659230192 P 8b9e878e72f64482bdd89f28ac427494: Neither blocks nor log segments found. Creating new log.
I20240502 02:02:13.741357  1667 tablet_bootstrap.cc:654] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25: Neither blocks nor log segments found. Creating new log.
I20240502 02:02:13.741724  1668 raft_consensus.cc:348] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:13.742012  1668 raft_consensus.cc:374] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:02:13.742132  1668 raft_consensus.cc:729] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0e4135c07ac4f2389ffe4e7379df71f, State: Initialized, Role: FOLLOWER
I20240502 02:02:13.742462  1668 consensus_queue.cc:260] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:13.745396  1668 ts_tablet_manager.cc:1398] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240502 02:02:13.745764  1668 tablet_bootstrap.cc:492] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f: Bootstrap starting.
I20240502 02:02:13.745956  1670 tablet_bootstrap.cc:492] T 11210087d9164d2c9c1303e659230192 P 8b9e878e72f64482bdd89f28ac427494: No bootstrap required, opened a new log
I20240502 02:02:13.746114  1670 ts_tablet_manager.cc:1367] T 11210087d9164d2c9c1303e659230192 P 8b9e878e72f64482bdd89f28ac427494: Time spent bootstrapping tablet: real 0.009s	user 0.002s	sys 0.004s
I20240502 02:02:13.746901  1670 raft_consensus.cc:348] T 11210087d9164d2c9c1303e659230192 P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:13.747126  1670 raft_consensus.cc:374] T 11210087d9164d2c9c1303e659230192 P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:02:13.747196  1670 raft_consensus.cc:729] T 11210087d9164d2c9c1303e659230192 P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b9e878e72f64482bdd89f28ac427494, State: Initialized, Role: FOLLOWER
I20240502 02:02:13.747551  1668 tablet_bootstrap.cc:654] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f: Neither blocks nor log segments found. Creating new log.
I20240502 02:02:13.747469  1670 consensus_queue.cc:260] T 11210087d9164d2c9c1303e659230192 P 8b9e878e72f64482bdd89f28ac427494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:13.748694  1670 ts_tablet_manager.cc:1398] T 11210087d9164d2c9c1303e659230192 P 8b9e878e72f64482bdd89f28ac427494: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240502 02:02:13.748988  1670 tablet_bootstrap.cc:492] T 884b0248d66f4575a7e8e3ead42c58c6 P 8b9e878e72f64482bdd89f28ac427494: Bootstrap starting.
I20240502 02:02:13.749418  1667 tablet_bootstrap.cc:492] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25: No bootstrap required, opened a new log
I20240502 02:02:13.749586  1667 ts_tablet_manager.cc:1367] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25: Time spent bootstrapping tablet: real 0.010s	user 0.003s	sys 0.006s
I20240502 02:02:13.750367  1667 raft_consensus.cc:348] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:13.750581  1667 raft_consensus.cc:374] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:02:13.750658  1667 raft_consensus.cc:729] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d37e50431f30463c872c5d819ddf2c25, State: Initialized, Role: FOLLOWER
I20240502 02:02:13.750947  1667 consensus_queue.cc:260] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:13.752249  1670 tablet_bootstrap.cc:654] T 884b0248d66f4575a7e8e3ead42c58c6 P 8b9e878e72f64482bdd89f28ac427494: Neither blocks nor log segments found. Creating new log.
I20240502 02:02:13.752270  1667 ts_tablet_manager.cc:1398] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.001s
I20240502 02:02:13.752565  1667 tablet_bootstrap.cc:492] T 53d5bc373eb14d44bf1dd07375f40f00 P d37e50431f30463c872c5d819ddf2c25: Bootstrap starting.
I20240502 02:02:13.754501  1667 tablet_bootstrap.cc:654] T 53d5bc373eb14d44bf1dd07375f40f00 P d37e50431f30463c872c5d819ddf2c25: Neither blocks nor log segments found. Creating new log.
I20240502 02:02:13.757403  1668 tablet_bootstrap.cc:492] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f: No bootstrap required, opened a new log
I20240502 02:02:13.757400  1667 tablet_bootstrap.cc:492] T 53d5bc373eb14d44bf1dd07375f40f00 P d37e50431f30463c872c5d819ddf2c25: No bootstrap required, opened a new log
I20240502 02:02:13.757436  1670 tablet_bootstrap.cc:492] T 884b0248d66f4575a7e8e3ead42c58c6 P 8b9e878e72f64482bdd89f28ac427494: No bootstrap required, opened a new log
I20240502 02:02:13.757560  1667 ts_tablet_manager.cc:1367] T 53d5bc373eb14d44bf1dd07375f40f00 P d37e50431f30463c872c5d819ddf2c25: Time spent bootstrapping tablet: real 0.005s	user 0.001s	sys 0.003s
I20240502 02:02:13.757567  1668 ts_tablet_manager.cc:1367] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f: Time spent bootstrapping tablet: real 0.012s	user 0.001s	sys 0.010s
I20240502 02:02:13.757587  1670 ts_tablet_manager.cc:1367] T 884b0248d66f4575a7e8e3ead42c58c6 P 8b9e878e72f64482bdd89f28ac427494: Time spent bootstrapping tablet: real 0.009s	user 0.003s	sys 0.004s
I20240502 02:02:13.758203  1670 raft_consensus.cc:348] T 884b0248d66f4575a7e8e3ead42c58c6 P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:13.758378  1670 raft_consensus.cc:374] T 884b0248d66f4575a7e8e3ead42c58c6 P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:02:13.758322  1667 raft_consensus.cc:348] T 53d5bc373eb14d44bf1dd07375f40f00 P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:13.758322  1668 raft_consensus.cc:348] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:13.758469  1670 raft_consensus.cc:729] T 884b0248d66f4575a7e8e3ead42c58c6 P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b9e878e72f64482bdd89f28ac427494, State: Initialized, Role: FOLLOWER
I20240502 02:02:13.758533  1668 raft_consensus.cc:374] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:02:13.758533  1667 raft_consensus.cc:374] T 53d5bc373eb14d44bf1dd07375f40f00 P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 02:02:13.758634  1668 raft_consensus.cc:729] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0e4135c07ac4f2389ffe4e7379df71f, State: Initialized, Role: FOLLOWER
I20240502 02:02:13.758702  1667 raft_consensus.cc:729] T 53d5bc373eb14d44bf1dd07375f40f00 P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d37e50431f30463c872c5d819ddf2c25, State: Initialized, Role: FOLLOWER
I20240502 02:02:13.758724  1670 consensus_queue.cc:260] T 884b0248d66f4575a7e8e3ead42c58c6 P 8b9e878e72f64482bdd89f28ac427494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:13.758874  1668 consensus_queue.cc:260] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:13.759294  1670 ts_tablet_manager.cc:1398] T 884b0248d66f4575a7e8e3ead42c58c6 P 8b9e878e72f64482bdd89f28ac427494: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 02:02:13.759441  1668 ts_tablet_manager.cc:1398] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 02:02:13.758949  1667 consensus_queue.cc:260] T 53d5bc373eb14d44bf1dd07375f40f00 P d37e50431f30463c872c5d819ddf2c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:13.760105  1667 ts_tablet_manager.cc:1398] T 53d5bc373eb14d44bf1dd07375f40f00 P d37e50431f30463c872c5d819ddf2c25: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 02:02:13.906816  1499 raft_consensus.cc:482] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 02:02:13.907075  1499 raft_consensus.cc:504] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:13.907754  1499 leader_election.cc:288] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129:33963), 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130:43631)
I20240502 02:02:13.908382  1018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "884b0248d66f4575a7e8e3ead42c58c6" candidate_uuid: "d37e50431f30463c872c5d819ddf2c25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" is_pre_election: true
I20240502 02:02:13.908408  1156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "884b0248d66f4575a7e8e3ead42c58c6" candidate_uuid: "d37e50431f30463c872c5d819ddf2c25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b9e878e72f64482bdd89f28ac427494" is_pre_election: true
I20240502 02:02:13.908628  1156 raft_consensus.cc:2454] T 884b0248d66f4575a7e8e3ead42c58c6 P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d37e50431f30463c872c5d819ddf2c25 in term 0.
I20240502 02:02:13.908617  1018 raft_consensus.cc:2454] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d37e50431f30463c872c5d819ddf2c25 in term 0.
I20240502 02:02:13.909104  1221 leader_election.cc:302] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0e4135c07ac4f2389ffe4e7379df71f, d37e50431f30463c872c5d819ddf2c25; no voters: 
I20240502 02:02:13.909420  1499 raft_consensus.cc:2781] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 02:02:13.909519  1499 raft_consensus.cc:482] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 02:02:13.909619  1499 raft_consensus.cc:3037] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:02:13.911558  1499 raft_consensus.cc:504] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:13.912217  1499 leader_election.cc:288] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [CANDIDATE]: Term 1 election: Requested vote from peers d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129:33963), 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130:43631)
I20240502 02:02:13.912796  1018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "884b0248d66f4575a7e8e3ead42c58c6" candidate_uuid: "d37e50431f30463c872c5d819ddf2c25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e4135c07ac4f2389ffe4e7379df71f"
I20240502 02:02:13.912801  1156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "884b0248d66f4575a7e8e3ead42c58c6" candidate_uuid: "d37e50431f30463c872c5d819ddf2c25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b9e878e72f64482bdd89f28ac427494"
I20240502 02:02:13.913031  1018 raft_consensus.cc:3037] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:02:13.913031  1156 raft_consensus.cc:3037] T 884b0248d66f4575a7e8e3ead42c58c6 P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:02:13.915831  1018 raft_consensus.cc:2454] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d37e50431f30463c872c5d819ddf2c25 in term 1.
I20240502 02:02:13.915831  1156 raft_consensus.cc:2454] T 884b0248d66f4575a7e8e3ead42c58c6 P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d37e50431f30463c872c5d819ddf2c25 in term 1.
I20240502 02:02:13.916355  1221 leader_election.cc:302] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0e4135c07ac4f2389ffe4e7379df71f, d37e50431f30463c872c5d819ddf2c25; no voters: 
I20240502 02:02:13.916635  1499 raft_consensus.cc:2781] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:02:13.916806  1499 raft_consensus.cc:686] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [term 1 LEADER]: Becoming Leader. State: Replica: d37e50431f30463c872c5d819ddf2c25, State: Running, Role: LEADER
I20240502 02:02:13.917085  1499 consensus_queue.cc:237] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:13.919834   871 catalog_manager.cc:5484] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 reported cstate change: term changed from 0 to 1, leader changed from <none> to d37e50431f30463c872c5d819ddf2c25 (127.25.109.131). New cstate: current_term: 1 leader_uuid: "d37e50431f30463c872c5d819ddf2c25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } health_report { overall_health: HEALTHY } } }
I20240502 02:02:13.999994  1498 raft_consensus.cc:482] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 02:02:14.000276  1498 raft_consensus.cc:504] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:14.000950  1498 leader_election.cc:288] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130:43631), d37e50431f30463c872c5d819ddf2c25 (127.25.109.131:37799)
I20240502 02:02:14.001644  1156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53d5bc373eb14d44bf1dd07375f40f00" candidate_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b9e878e72f64482bdd89f28ac427494" is_pre_election: true
I20240502 02:02:14.001649  1288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53d5bc373eb14d44bf1dd07375f40f00" candidate_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d37e50431f30463c872c5d819ddf2c25" is_pre_election: true
I20240502 02:02:14.001935  1288 raft_consensus.cc:2454] T 53d5bc373eb14d44bf1dd07375f40f00 P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0e4135c07ac4f2389ffe4e7379df71f in term 0.
I20240502 02:02:14.001933  1156 raft_consensus.cc:2454] T 53d5bc373eb14d44bf1dd07375f40f00 P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0e4135c07ac4f2389ffe4e7379df71f in term 0.
I20240502 02:02:14.002453   951 leader_election.cc:302] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8b9e878e72f64482bdd89f28ac427494, d0e4135c07ac4f2389ffe4e7379df71f, d37e50431f30463c872c5d819ddf2c25; no voters: 
I20240502 02:02:14.002768  1498 raft_consensus.cc:2781] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 02:02:14.002869  1498 raft_consensus.cc:482] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 02:02:14.002979  1498 raft_consensus.cc:3037] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:02:14.005053  1498 raft_consensus.cc:504] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:14.005718  1498 leader_election.cc:288] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [CANDIDATE]: Term 1 election: Requested vote from peers 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130:43631), d37e50431f30463c872c5d819ddf2c25 (127.25.109.131:37799)
I20240502 02:02:14.006206  1288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53d5bc373eb14d44bf1dd07375f40f00" candidate_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d37e50431f30463c872c5d819ddf2c25"
I20240502 02:02:14.006199  1156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53d5bc373eb14d44bf1dd07375f40f00" candidate_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b9e878e72f64482bdd89f28ac427494"
I20240502 02:02:14.006445  1156 raft_consensus.cc:3037] T 53d5bc373eb14d44bf1dd07375f40f00 P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:02:14.006453  1288 raft_consensus.cc:3037] T 53d5bc373eb14d44bf1dd07375f40f00 P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:02:14.009209  1288 raft_consensus.cc:2454] T 53d5bc373eb14d44bf1dd07375f40f00 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0e4135c07ac4f2389ffe4e7379df71f in term 1.
I20240502 02:02:14.009209  1156 raft_consensus.cc:2454] T 53d5bc373eb14d44bf1dd07375f40f00 P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0e4135c07ac4f2389ffe4e7379df71f in term 1.
I20240502 02:02:14.009709   953 leader_election.cc:302] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b9e878e72f64482bdd89f28ac427494, d0e4135c07ac4f2389ffe4e7379df71f; no voters: 
I20240502 02:02:14.010020  1498 raft_consensus.cc:2781] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:02:14.010222  1498 raft_consensus.cc:686] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 LEADER]: Becoming Leader. State: Replica: d0e4135c07ac4f2389ffe4e7379df71f, State: Running, Role: LEADER
I20240502 02:02:14.010522  1498 consensus_queue.cc:237] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:14.013417   871 catalog_manager.cc:5484] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f reported cstate change: term changed from 0 to 1, leader changed from <none> to d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129). New cstate: current_term: 1 leader_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } health_report { overall_health: UNKNOWN } } }
I20240502 02:02:14.026645  1498 raft_consensus.cc:482] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 02:02:14.026859  1498 raft_consensus.cc:504] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:14.027537  1498 leader_election.cc:288] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130:43631), d37e50431f30463c872c5d819ddf2c25 (127.25.109.131:37799)
I20240502 02:02:14.028239  1288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11210087d9164d2c9c1303e659230192" candidate_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d37e50431f30463c872c5d819ddf2c25" is_pre_election: true
I20240502 02:02:14.028240  1156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11210087d9164d2c9c1303e659230192" candidate_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b9e878e72f64482bdd89f28ac427494" is_pre_election: true
I20240502 02:02:14.028515  1156 raft_consensus.cc:2454] T 11210087d9164d2c9c1303e659230192 P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0e4135c07ac4f2389ffe4e7379df71f in term 0.
I20240502 02:02:14.028515  1288 raft_consensus.cc:2454] T 11210087d9164d2c9c1303e659230192 P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0e4135c07ac4f2389ffe4e7379df71f in term 0.
I20240502 02:02:14.028972   951 leader_election.cc:302] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0e4135c07ac4f2389ffe4e7379df71f, d37e50431f30463c872c5d819ddf2c25; no voters: 
I20240502 02:02:14.029261  1498 raft_consensus.cc:2781] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 02:02:14.029377  1498 raft_consensus.cc:482] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 02:02:14.029475  1498 raft_consensus.cc:3037] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:02:14.031344  1498 raft_consensus.cc:504] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:14.031946  1498 leader_election.cc:288] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f [CANDIDATE]: Term 1 election: Requested vote from peers 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130:43631), d37e50431f30463c872c5d819ddf2c25 (127.25.109.131:37799)
I20240502 02:02:14.032517  1288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11210087d9164d2c9c1303e659230192" candidate_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d37e50431f30463c872c5d819ddf2c25"
I20240502 02:02:14.032517  1156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11210087d9164d2c9c1303e659230192" candidate_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b9e878e72f64482bdd89f28ac427494"
I20240502 02:02:14.032745  1288 raft_consensus.cc:3037] T 11210087d9164d2c9c1303e659230192 P d37e50431f30463c872c5d819ddf2c25 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:02:14.032765  1156 raft_consensus.cc:3037] T 11210087d9164d2c9c1303e659230192 P 8b9e878e72f64482bdd89f28ac427494 [term 0 FOLLOWER]: Advancing to term 1
I20240502 02:02:14.035631  1156 raft_consensus.cc:2454] T 11210087d9164d2c9c1303e659230192 P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0e4135c07ac4f2389ffe4e7379df71f in term 1.
I20240502 02:02:14.035637  1288 raft_consensus.cc:2454] T 11210087d9164d2c9c1303e659230192 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0e4135c07ac4f2389ffe4e7379df71f in term 1.
I20240502 02:02:14.036167   951 leader_election.cc:302] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0e4135c07ac4f2389ffe4e7379df71f, d37e50431f30463c872c5d819ddf2c25; no voters: 
I20240502 02:02:14.036479  1498 raft_consensus.cc:2781] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 02:02:14.036623  1498 raft_consensus.cc:686] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 LEADER]: Becoming Leader. State: Replica: d0e4135c07ac4f2389ffe4e7379df71f, State: Running, Role: LEADER
I20240502 02:02:14.036916  1498 consensus_queue.cc:237] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } }
I20240502 02:02:14.039626   871 catalog_manager.cc:5484] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f reported cstate change: term changed from 0 to 1, leader changed from <none> to d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129). New cstate: current_term: 1 leader_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } health_report { overall_health: UNKNOWN } } }
I20240502 02:02:14.353135  1499 consensus_queue.cc:1035] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:02:14.362205  1499 consensus_queue.cc:1035] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 02:02:14.432157  1682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:02:14.432538  1682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:02:14.440734  1682 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 02:02:14.460114  1498 consensus_queue.cc:1035] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:02:14.464725  1498 consensus_queue.cc:1035] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240502 02:02:14.559782  1596 consensus_queue.cc:1035] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 02:02:14.572166  1156 raft_consensus.cc:1261] T 53d5bc373eb14d44bf1dd07375f40f00 P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 02:02:14.573163  1498 consensus_queue.cc:1035] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 189.988 ms
  time per row: 0.0949938 ms
I20240502 02:02:14.778985 26038 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
/tmp/dist-test-taskb4pM1q/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.109.132:44441
--local_ip_for_outbound_sockets=127.25.109.132
--tserver_master_addrs=127.25.109.190:33343
--webserver_port=34929
--webserver_interface=127.25.109.132
--builtin_ntp_servers=127.25.109.148:45821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 02:02:15.135303  1742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 02:02:15.135716  1742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 02:02:15.135876  1742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 02:02:15.143959  1742 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 02:02:15.144109  1742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 02:02:15.144272  1742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.109.132
I20240502 02:02:15.153781  1742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.109.148:45821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.109.132:44441
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.25.109.132
--webserver_port=34929
--tserver_master_addrs=127.25.109.190:33343
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.109.132
--log_dir=/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 02:02:15.155316  1742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 02:02:15.157140  1742 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 02:02:15.173275  1748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:02:15.173728  1747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 02:02:15.174119  1750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 02:02:15.174614  1742 server_base.cc:1034] running on GCE node
I20240502 02:02:15.175318  1742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 02:02:15.176357  1742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 02:02:15.177585  1742 hybrid_clock.cc:648] HybridClock initialized: now 1714615335177516 us; error 72 us; skew 500 ppm
I20240502 02:02:15.177974  1742 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 02:02:15.180717  1742 webserver.cc:460] Webserver started at http://127.25.109.132:34929/ using document root <none> and password file <none>
I20240502 02:02:15.181481  1742 fs_manager.cc:359] Metadata directory not provided
I20240502 02:02:15.181622  1742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 02:02:15.187645  1742 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.003s
I20240502 02:02:15.190804  1756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:02:15.192818  1742 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
I20240502 02:02:15.193006  1742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "eb5a2343514740caa85ff0228a535123"
format_stamp: "Formatted at 2024-05-02 02:02:07 on dist-test-slave-kj4b"
I20240502 02:02:15.194303  1742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 02:02:15.217859  1742 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 02:02:15.218721  1742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 02:02:15.218999  1742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 02:02:15.220005  1742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 02:02:15.221565  1742 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 02:02:15.221643  1742 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:02:15.221771  1742 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 02:02:15.221822  1742 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 02:02:15.252321  1742 rpc_server.cc:307] RPC server started. Bound to: 127.25.109.132:44441
I20240502 02:02:15.252375  1870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.109.132:44441 every 8 connection(s)
I20240502 02:02:15.253736  1742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 02:02:15.255435 26038 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu as pid 1742
I20240502 02:02:15.268152  1871 heartbeater.cc:347] Connected to a master server at 127.25.109.190:33343
I20240502 02:02:15.268494  1871 heartbeater.cc:464] Registering TS with master...
I20240502 02:02:15.269399  1871 heartbeater.cc:511] Master 127.25.109.190:33343 requested a full tablet report, sending...
I20240502 02:02:15.271200   873 ts_manager.cc:194] Re-registered known tserver with Master: eb5a2343514740caa85ff0228a535123 (127.25.109.132:44441)
I20240502 02:02:15.272189   873 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.109.132:39889
I20240502 02:02:15.756467  1135 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:02:15.775745   998 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:15.784384  1804 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:02:15.784826  1269 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 02:02:15.948202  1135 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:02:15.949753   998 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:15.957715  1269 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 02:02:15.967294  1804 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:02:16.137099  1135 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:02:16.140514   998 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:16.140872  1804 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:02:16.141547  1269 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 02:02:16.275506  1871 heartbeater.cc:503] Master 127.25.109.190:33343 was elected leader, sending a full tablet report...
I20240502 02:02:16.307209  1269 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 02:02:16.307281  1135 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:02:16.307565   998 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:16.307847  1804 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:02:16.384485  1018 consensus_queue.cc:237] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } }
I20240502 02:02:16.387156  1156 raft_consensus.cc:1261] T 53d5bc373eb14d44bf1dd07375f40f00 P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 02:02:16.387228  1289 raft_consensus.cc:1261] T 53d5bc373eb14d44bf1dd07375f40f00 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 02:02:16.387948  1699 consensus_queue.cc:1035] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:02:16.388350  1975 consensus_queue.cc:1035] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:02:16.391788  1699 raft_consensus.cc:2932] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER eb5a2343514740caa85ff0228a535123 (127.25.109.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } } }
I20240502 02:02:16.396257  1289 raft_consensus.cc:2932] T 53d5bc373eb14d44bf1dd07375f40f00 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER eb5a2343514740caa85ff0228a535123 (127.25.109.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } } }
I20240502 02:02:16.396452  1156 raft_consensus.cc:2932] T 53d5bc373eb14d44bf1dd07375f40f00 P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER eb5a2343514740caa85ff0228a535123 (127.25.109.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } } }
W20240502 02:02:16.400940   954 consensus_peers.cc:485] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f -> Peer eb5a2343514740caa85ff0228a535123 (127.25.109.132:44441): Couldn't send request to peer eb5a2343514740caa85ff0228a535123. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 53d5bc373eb14d44bf1dd07375f40f00. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:02:16.402359   873 catalog_manager.cc:5484] T 53d5bc373eb14d44bf1dd07375f40f00 P d37e50431f30463c872c5d819ddf2c25 reported cstate change: config changed from index -1 to 4, NON_VOTER eb5a2343514740caa85ff0228a535123 (127.25.109.132) added. New cstate: current_term: 1 leader_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } } }
I20240502 02:02:16.426220  1289 consensus_queue.cc:237] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { replace: true } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } }
I20240502 02:02:16.429148  1018 raft_consensus.cc:1261] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Refusing update from remote peer d37e50431f30463c872c5d819ddf2c25: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 02:02:16.429381  1156 raft_consensus.cc:1261] T b1005c925b954ca39a9ee6f2677b3a1f P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Refusing update from remote peer d37e50431f30463c872c5d819ddf2c25: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 02:02:16.429850  2028 consensus_queue.cc:1035] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:02:16.430289  1731 consensus_queue.cc:1035] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:02:16.433679  1731 raft_consensus.cc:2932] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER eb5a2343514740caa85ff0228a535123 (127.25.109.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { replace: true } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } } }
I20240502 02:02:16.434237  1156 raft_consensus.cc:2932] T b1005c925b954ca39a9ee6f2677b3a1f P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER eb5a2343514740caa85ff0228a535123 (127.25.109.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { replace: true } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } } }
I20240502 02:02:16.436547  1018 raft_consensus.cc:2932] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER eb5a2343514740caa85ff0228a535123 (127.25.109.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { replace: true } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } } }
W20240502 02:02:16.437188  1223 consensus_peers.cc:485] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 -> Peer eb5a2343514740caa85ff0228a535123 (127.25.109.132:44441): Couldn't send request to peer eb5a2343514740caa85ff0228a535123. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b1005c925b954ca39a9ee6f2677b3a1f. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:02:16.440433   866 catalog_manager.cc:5484] T b1005c925b954ca39a9ee6f2677b3a1f P 8b9e878e72f64482bdd89f28ac427494 reported cstate change: config changed from index -1 to 4, NON_VOTER eb5a2343514740caa85ff0228a535123 (127.25.109.132) added. New cstate: current_term: 1 leader_uuid: "d37e50431f30463c872c5d819ddf2c25" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { replace: true } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } } }
I20240502 02:02:16.455487  1156 consensus_queue.cc:237] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } }
I20240502 02:02:16.458330  1018 raft_consensus.cc:1261] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Refusing update from remote peer 8b9e878e72f64482bdd89f28ac427494: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 02:02:16.458556  1289 raft_consensus.cc:1261] T b80d547302ec453aaf80fa6b64eebc13 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Refusing update from remote peer 8b9e878e72f64482bdd89f28ac427494: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 02:02:16.459080  2040 consensus_queue.cc:1035] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:02:16.459405  1658 consensus_queue.cc:1035] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:02:16.462529  1658 raft_consensus.cc:2932] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER eb5a2343514740caa85ff0228a535123 (127.25.109.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } } }
I20240502 02:02:16.463222  1289 raft_consensus.cc:2932] T b80d547302ec453aaf80fa6b64eebc13 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER eb5a2343514740caa85ff0228a535123 (127.25.109.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } } }
W20240502 02:02:16.467424  1089 consensus_peers.cc:485] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 -> Peer eb5a2343514740caa85ff0228a535123 (127.25.109.132:44441): Couldn't send request to peer eb5a2343514740caa85ff0228a535123. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b80d547302ec453aaf80fa6b64eebc13. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:02:16.471186   866 catalog_manager.cc:5484] T b80d547302ec453aaf80fa6b64eebc13 P d37e50431f30463c872c5d819ddf2c25 reported cstate change: config changed from index -1 to 4, NON_VOTER eb5a2343514740caa85ff0228a535123 (127.25.109.132) added. New cstate: current_term: 1 leader_uuid: "8b9e878e72f64482bdd89f28ac427494" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } } }
I20240502 02:02:16.477964  1018 raft_consensus.cc:2932] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER eb5a2343514740caa85ff0228a535123 (127.25.109.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } } }
I20240502 02:02:16.491562  1156 consensus_queue.cc:237] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { replace: true } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } }
I20240502 02:02:16.494124  1018 raft_consensus.cc:1261] T 9e7759be006b4a8186aca434c48949b2 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Refusing update from remote peer 8b9e878e72f64482bdd89f28ac427494: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 02:02:16.494136  1289 raft_consensus.cc:1261] T 9e7759be006b4a8186aca434c48949b2 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Refusing update from remote peer 8b9e878e72f64482bdd89f28ac427494: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 02:02:16.494310  1089 consensus_peers.cc:485] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 -> Peer eb5a2343514740caa85ff0228a535123 (127.25.109.132:44441): Couldn't send request to peer eb5a2343514740caa85ff0228a535123. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9e7759be006b4a8186aca434c48949b2. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:02:16.494719  2045 consensus_queue.cc:1035] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:02:16.495064  2040 consensus_queue.cc:1035] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:02:16.497900  1658 raft_consensus.cc:2932] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER eb5a2343514740caa85ff0228a535123 (127.25.109.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { replace: true } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } } }
I20240502 02:02:16.498524  1289 raft_consensus.cc:2932] T 9e7759be006b4a8186aca434c48949b2 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER eb5a2343514740caa85ff0228a535123 (127.25.109.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { replace: true } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } } }
I20240502 02:02:16.498662  1018 raft_consensus.cc:2932] T 9e7759be006b4a8186aca434c48949b2 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER eb5a2343514740caa85ff0228a535123 (127.25.109.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { replace: true } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } } }
I20240502 02:02:16.504088   866 catalog_manager.cc:5484] T 9e7759be006b4a8186aca434c48949b2 P d0e4135c07ac4f2389ffe4e7379df71f reported cstate change: config changed from index -1 to 4, NON_VOTER eb5a2343514740caa85ff0228a535123 (127.25.109.132) added. New cstate: current_term: 1 leader_uuid: "8b9e878e72f64482bdd89f28ac427494" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { replace: true } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } } }
I20240502 02:02:16.516906  1018 consensus_queue.cc:237] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } }
I20240502 02:02:16.519121  1289 raft_consensus.cc:1261] T f55670daadbc4a7f88971471629c793f P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 02:02:16.519235  1156 raft_consensus.cc:1261] T f55670daadbc4a7f88971471629c793f P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 02:02:16.519382   954 consensus_peers.cc:485] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f -> Peer eb5a2343514740caa85ff0228a535123 (127.25.109.132:44441): Couldn't send request to peer eb5a2343514740caa85ff0228a535123. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f55670daadbc4a7f88971471629c793f. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:02:16.519783  2014 consensus_queue.cc:1035] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:02:16.520151  1699 consensus_queue.cc:1035] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:02:16.522683  1975 raft_consensus.cc:2932] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER eb5a2343514740caa85ff0228a535123 (127.25.109.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } } }
I20240502 02:02:16.523415  1156 raft_consensus.cc:2932] T f55670daadbc4a7f88971471629c793f P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER eb5a2343514740caa85ff0228a535123 (127.25.109.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } } }
I20240502 02:02:16.523344  1289 raft_consensus.cc:2932] T f55670daadbc4a7f88971471629c793f P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER eb5a2343514740caa85ff0228a535123 (127.25.109.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } } }
I20240502 02:02:16.528342   873 catalog_manager.cc:5484] T f55670daadbc4a7f88971471629c793f P d37e50431f30463c872c5d819ddf2c25 reported cstate change: config changed from index -1 to 4, NON_VOTER eb5a2343514740caa85ff0228a535123 (127.25.109.132) added. New cstate: current_term: 1 leader_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } } }
I20240502 02:02:16.870965  2113 ts_tablet_manager.cc:902] T b80d547302ec453aaf80fa6b64eebc13 P eb5a2343514740caa85ff0228a535123: Initiating tablet copy from peer 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130:43631)
I20240502 02:02:16.872699  2113 tablet_copy_client.cc:308] T b80d547302ec453aaf80fa6b64eebc13 P eb5a2343514740caa85ff0228a535123: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.130:43631
I20240502 02:02:16.879022  1176 tablet_copy_service.cc:140] P 8b9e878e72f64482bdd89f28ac427494: Received BeginTabletCopySession request for tablet b80d547302ec453aaf80fa6b64eebc13 from peer eb5a2343514740caa85ff0228a535123 ({username='slave'} at 127.25.109.132:52325)
I20240502 02:02:16.879271  1176 tablet_copy_service.cc:161] P 8b9e878e72f64482bdd89f28ac427494: Beginning new tablet copy session on tablet b80d547302ec453aaf80fa6b64eebc13 from peer eb5a2343514740caa85ff0228a535123 at {username='slave'} at 127.25.109.132:52325: session id = eb5a2343514740caa85ff0228a535123-b80d547302ec453aaf80fa6b64eebc13
I20240502 02:02:16.881917  1176 tablet_copy_source_session.cc:205] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 02:02:16.886513  2113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b80d547302ec453aaf80fa6b64eebc13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:16.897003  2113 tablet_copy_client.cc:783] T b80d547302ec453aaf80fa6b64eebc13 P eb5a2343514740caa85ff0228a535123: tablet copy: Starting download of 0 data blocks...
I20240502 02:02:16.897356  2113 tablet_copy_client.cc:647] T b80d547302ec453aaf80fa6b64eebc13 P eb5a2343514740caa85ff0228a535123: tablet copy: Starting download of 1 WAL segments...
I20240502 02:02:16.899739  2113 tablet_copy_client.cc:523] T b80d547302ec453aaf80fa6b64eebc13 P eb5a2343514740caa85ff0228a535123: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:02:16.903111  2113 tablet_bootstrap.cc:492] T b80d547302ec453aaf80fa6b64eebc13 P eb5a2343514740caa85ff0228a535123: Bootstrap starting.
I20240502 02:02:16.912775  2113 log.cc:824] T b80d547302ec453aaf80fa6b64eebc13 P eb5a2343514740caa85ff0228a535123: Log is configured to *not* fsync() on all Append() calls
I20240502 02:02:16.943822  2113 tablet_bootstrap.cc:492] T b80d547302ec453aaf80fa6b64eebc13 P eb5a2343514740caa85ff0228a535123: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:02:16.944393  2113 tablet_bootstrap.cc:492] T b80d547302ec453aaf80fa6b64eebc13 P eb5a2343514740caa85ff0228a535123: Bootstrap complete.
I20240502 02:02:16.944867  2113 ts_tablet_manager.cc:1367] T b80d547302ec453aaf80fa6b64eebc13 P eb5a2343514740caa85ff0228a535123: Time spent bootstrapping tablet: real 0.042s	user 0.039s	sys 0.004s
I20240502 02:02:16.949514  2113 raft_consensus.cc:348] T b80d547302ec453aaf80fa6b64eebc13 P eb5a2343514740caa85ff0228a535123 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } }
I20240502 02:02:16.949914  2113 raft_consensus.cc:729] T b80d547302ec453aaf80fa6b64eebc13 P eb5a2343514740caa85ff0228a535123 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb5a2343514740caa85ff0228a535123, State: Initialized, Role: LEARNER
I20240502 02:02:16.950389  2113 consensus_queue.cc:260] T b80d547302ec453aaf80fa6b64eebc13 P eb5a2343514740caa85ff0228a535123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } }
I20240502 02:02:16.951797  2113 ts_tablet_manager.cc:1398] T b80d547302ec453aaf80fa6b64eebc13 P eb5a2343514740caa85ff0228a535123: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20240502 02:02:16.952803  1176 tablet_copy_service.cc:342] P 8b9e878e72f64482bdd89f28ac427494: Request end of tablet copy session eb5a2343514740caa85ff0228a535123-b80d547302ec453aaf80fa6b64eebc13 received from {username='slave'} at 127.25.109.132:52325
I20240502 02:02:16.952965  1176 tablet_copy_service.cc:433] P 8b9e878e72f64482bdd89f28ac427494: ending tablet copy session eb5a2343514740caa85ff0228a535123-b80d547302ec453aaf80fa6b64eebc13 on tablet b80d547302ec453aaf80fa6b64eebc13 with peer eb5a2343514740caa85ff0228a535123
I20240502 02:02:16.958786  2113 ts_tablet_manager.cc:902] T b1005c925b954ca39a9ee6f2677b3a1f P eb5a2343514740caa85ff0228a535123: Initiating tablet copy from peer d37e50431f30463c872c5d819ddf2c25 (127.25.109.131:37799)
I20240502 02:02:16.959785  2113 tablet_copy_client.cc:308] T b1005c925b954ca39a9ee6f2677b3a1f P eb5a2343514740caa85ff0228a535123: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.131:37799
I20240502 02:02:16.964321  2125 ts_tablet_manager.cc:902] T 53d5bc373eb14d44bf1dd07375f40f00 P eb5a2343514740caa85ff0228a535123: Initiating tablet copy from peer d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129:33963)
I20240502 02:02:16.966181  1309 tablet_copy_service.cc:140] P d37e50431f30463c872c5d819ddf2c25: Received BeginTabletCopySession request for tablet b1005c925b954ca39a9ee6f2677b3a1f from peer eb5a2343514740caa85ff0228a535123 ({username='slave'} at 127.25.109.132:43379)
I20240502 02:02:16.966400  2125 tablet_copy_client.cc:308] T 53d5bc373eb14d44bf1dd07375f40f00 P eb5a2343514740caa85ff0228a535123: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.129:33963
I20240502 02:02:16.966430  1309 tablet_copy_service.cc:161] P d37e50431f30463c872c5d819ddf2c25: Beginning new tablet copy session on tablet b1005c925b954ca39a9ee6f2677b3a1f from peer eb5a2343514740caa85ff0228a535123 at {username='slave'} at 127.25.109.132:43379: session id = eb5a2343514740caa85ff0228a535123-b1005c925b954ca39a9ee6f2677b3a1f
I20240502 02:02:16.969810  1309 tablet_copy_source_session.cc:205] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 02:02:16.971546  2113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1005c925b954ca39a9ee6f2677b3a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:16.972694  1040 tablet_copy_service.cc:140] P d0e4135c07ac4f2389ffe4e7379df71f: Received BeginTabletCopySession request for tablet 53d5bc373eb14d44bf1dd07375f40f00 from peer eb5a2343514740caa85ff0228a535123 ({username='slave'} at 127.25.109.132:40517)
I20240502 02:02:16.972900  1040 tablet_copy_service.cc:161] P d0e4135c07ac4f2389ffe4e7379df71f: Beginning new tablet copy session on tablet 53d5bc373eb14d44bf1dd07375f40f00 from peer eb5a2343514740caa85ff0228a535123 at {username='slave'} at 127.25.109.132:40517: session id = eb5a2343514740caa85ff0228a535123-53d5bc373eb14d44bf1dd07375f40f00
I20240502 02:02:16.975769  1040 tablet_copy_source_session.cc:205] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 02:02:16.976608  2113 tablet_copy_client.cc:783] T b1005c925b954ca39a9ee6f2677b3a1f P eb5a2343514740caa85ff0228a535123: tablet copy: Starting download of 0 data blocks...
I20240502 02:02:16.976923  2113 tablet_copy_client.cc:647] T b1005c925b954ca39a9ee6f2677b3a1f P eb5a2343514740caa85ff0228a535123: tablet copy: Starting download of 1 WAL segments...
I20240502 02:02:16.977653  2125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53d5bc373eb14d44bf1dd07375f40f00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:16.979504  2113 tablet_copy_client.cc:523] T b1005c925b954ca39a9ee6f2677b3a1f P eb5a2343514740caa85ff0228a535123: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:02:16.982147  2113 tablet_bootstrap.cc:492] T b1005c925b954ca39a9ee6f2677b3a1f P eb5a2343514740caa85ff0228a535123: Bootstrap starting.
I20240502 02:02:16.982586  2125 tablet_copy_client.cc:783] T 53d5bc373eb14d44bf1dd07375f40f00 P eb5a2343514740caa85ff0228a535123: tablet copy: Starting download of 0 data blocks...
I20240502 02:02:16.982851  2125 tablet_copy_client.cc:647] T 53d5bc373eb14d44bf1dd07375f40f00 P eb5a2343514740caa85ff0228a535123: tablet copy: Starting download of 1 WAL segments...
I20240502 02:02:16.984902  2125 tablet_copy_client.cc:523] T 53d5bc373eb14d44bf1dd07375f40f00 P eb5a2343514740caa85ff0228a535123: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:02:16.988525  2125 tablet_bootstrap.cc:492] T 53d5bc373eb14d44bf1dd07375f40f00 P eb5a2343514740caa85ff0228a535123: Bootstrap starting.
I20240502 02:02:17.017731  2113 tablet_bootstrap.cc:492] T b1005c925b954ca39a9ee6f2677b3a1f P eb5a2343514740caa85ff0228a535123: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:02:17.018154  2113 tablet_bootstrap.cc:492] T b1005c925b954ca39a9ee6f2677b3a1f P eb5a2343514740caa85ff0228a535123: Bootstrap complete.
I20240502 02:02:17.018416  2113 ts_tablet_manager.cc:1367] T b1005c925b954ca39a9ee6f2677b3a1f P eb5a2343514740caa85ff0228a535123: Time spent bootstrapping tablet: real 0.036s	user 0.028s	sys 0.007s
I20240502 02:02:17.019282  2113 raft_consensus.cc:348] T b1005c925b954ca39a9ee6f2677b3a1f P eb5a2343514740caa85ff0228a535123 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { replace: true } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } }
I20240502 02:02:17.019515  2113 raft_consensus.cc:729] T b1005c925b954ca39a9ee6f2677b3a1f P eb5a2343514740caa85ff0228a535123 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb5a2343514740caa85ff0228a535123, State: Initialized, Role: LEARNER
I20240502 02:02:17.019724  2113 consensus_queue.cc:260] T b1005c925b954ca39a9ee6f2677b3a1f P eb5a2343514740caa85ff0228a535123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { replace: true } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } }
I20240502 02:02:17.020318  2113 ts_tablet_manager.cc:1398] T b1005c925b954ca39a9ee6f2677b3a1f P eb5a2343514740caa85ff0228a535123: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240502 02:02:17.021464  1309 tablet_copy_service.cc:342] P d37e50431f30463c872c5d819ddf2c25: Request end of tablet copy session eb5a2343514740caa85ff0228a535123-b1005c925b954ca39a9ee6f2677b3a1f received from {username='slave'} at 127.25.109.132:43379
I20240502 02:02:17.021678  1309 tablet_copy_service.cc:433] P d37e50431f30463c872c5d819ddf2c25: ending tablet copy session eb5a2343514740caa85ff0228a535123-b1005c925b954ca39a9ee6f2677b3a1f on tablet b1005c925b954ca39a9ee6f2677b3a1f with peer eb5a2343514740caa85ff0228a535123
I20240502 02:02:17.022058  2125 tablet_bootstrap.cc:492] T 53d5bc373eb14d44bf1dd07375f40f00 P eb5a2343514740caa85ff0228a535123: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:02:17.022498  2125 tablet_bootstrap.cc:492] T 53d5bc373eb14d44bf1dd07375f40f00 P eb5a2343514740caa85ff0228a535123: Bootstrap complete.
I20240502 02:02:17.022984  2125 ts_tablet_manager.cc:1367] T 53d5bc373eb14d44bf1dd07375f40f00 P eb5a2343514740caa85ff0228a535123: Time spent bootstrapping tablet: real 0.034s	user 0.024s	sys 0.008s
I20240502 02:02:17.023825  2125 raft_consensus.cc:348] T 53d5bc373eb14d44bf1dd07375f40f00 P eb5a2343514740caa85ff0228a535123 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } }
I20240502 02:02:17.024106  2125 raft_consensus.cc:729] T 53d5bc373eb14d44bf1dd07375f40f00 P eb5a2343514740caa85ff0228a535123 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb5a2343514740caa85ff0228a535123, State: Initialized, Role: LEARNER
I20240502 02:02:17.024298  2125 consensus_queue.cc:260] T 53d5bc373eb14d44bf1dd07375f40f00 P eb5a2343514740caa85ff0228a535123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } }
I20240502 02:02:17.024744  2125 ts_tablet_manager.cc:1398] T 53d5bc373eb14d44bf1dd07375f40f00 P eb5a2343514740caa85ff0228a535123: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 02:02:17.025625  1040 tablet_copy_service.cc:342] P d0e4135c07ac4f2389ffe4e7379df71f: Request end of tablet copy session eb5a2343514740caa85ff0228a535123-53d5bc373eb14d44bf1dd07375f40f00 received from {username='slave'} at 127.25.109.132:40517
I20240502 02:02:17.025830  1040 tablet_copy_service.cc:433] P d0e4135c07ac4f2389ffe4e7379df71f: ending tablet copy session eb5a2343514740caa85ff0228a535123-53d5bc373eb14d44bf1dd07375f40f00 on tablet 53d5bc373eb14d44bf1dd07375f40f00 with peer eb5a2343514740caa85ff0228a535123
I20240502 02:02:17.066481  2125 ts_tablet_manager.cc:902] T f55670daadbc4a7f88971471629c793f P eb5a2343514740caa85ff0228a535123: Initiating tablet copy from peer d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129:33963)
I20240502 02:02:17.067754  2125 tablet_copy_client.cc:308] T f55670daadbc4a7f88971471629c793f P eb5a2343514740caa85ff0228a535123: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.129:33963
I20240502 02:02:17.068511  1040 tablet_copy_service.cc:140] P d0e4135c07ac4f2389ffe4e7379df71f: Received BeginTabletCopySession request for tablet f55670daadbc4a7f88971471629c793f from peer eb5a2343514740caa85ff0228a535123 ({username='slave'} at 127.25.109.132:40517)
I20240502 02:02:17.068689  1040 tablet_copy_service.cc:161] P d0e4135c07ac4f2389ffe4e7379df71f: Beginning new tablet copy session on tablet f55670daadbc4a7f88971471629c793f from peer eb5a2343514740caa85ff0228a535123 at {username='slave'} at 127.25.109.132:40517: session id = eb5a2343514740caa85ff0228a535123-f55670daadbc4a7f88971471629c793f
I20240502 02:02:17.070817  1040 tablet_copy_source_session.cc:205] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 02:02:17.072089  2125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f55670daadbc4a7f88971471629c793f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:17.076905  2125 tablet_copy_client.cc:783] T f55670daadbc4a7f88971471629c793f P eb5a2343514740caa85ff0228a535123: tablet copy: Starting download of 0 data blocks...
I20240502 02:02:17.077139  2125 tablet_copy_client.cc:647] T f55670daadbc4a7f88971471629c793f P eb5a2343514740caa85ff0228a535123: tablet copy: Starting download of 1 WAL segments...
I20240502 02:02:17.078668  2125 tablet_copy_client.cc:523] T f55670daadbc4a7f88971471629c793f P eb5a2343514740caa85ff0228a535123: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:02:17.081076  2125 tablet_bootstrap.cc:492] T f55670daadbc4a7f88971471629c793f P eb5a2343514740caa85ff0228a535123: Bootstrap starting.
I20240502 02:02:17.100356  2113 ts_tablet_manager.cc:902] T 9e7759be006b4a8186aca434c48949b2 P eb5a2343514740caa85ff0228a535123: Initiating tablet copy from peer 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130:43631)
I20240502 02:02:17.102188  2113 tablet_copy_client.cc:308] T 9e7759be006b4a8186aca434c48949b2 P eb5a2343514740caa85ff0228a535123: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.130:43631
I20240502 02:02:17.102981  1176 tablet_copy_service.cc:140] P 8b9e878e72f64482bdd89f28ac427494: Received BeginTabletCopySession request for tablet 9e7759be006b4a8186aca434c48949b2 from peer eb5a2343514740caa85ff0228a535123 ({username='slave'} at 127.25.109.132:52325)
I20240502 02:02:17.103161  1176 tablet_copy_service.cc:161] P 8b9e878e72f64482bdd89f28ac427494: Beginning new tablet copy session on tablet 9e7759be006b4a8186aca434c48949b2 from peer eb5a2343514740caa85ff0228a535123 at {username='slave'} at 127.25.109.132:52325: session id = eb5a2343514740caa85ff0228a535123-9e7759be006b4a8186aca434c48949b2
I20240502 02:02:17.105247  1176 tablet_copy_source_session.cc:205] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 02:02:17.106446  2113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e7759be006b4a8186aca434c48949b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:17.111160  2113 tablet_copy_client.cc:783] T 9e7759be006b4a8186aca434c48949b2 P eb5a2343514740caa85ff0228a535123: tablet copy: Starting download of 0 data blocks...
I20240502 02:02:17.111430  2113 tablet_copy_client.cc:647] T 9e7759be006b4a8186aca434c48949b2 P eb5a2343514740caa85ff0228a535123: tablet copy: Starting download of 1 WAL segments...
I20240502 02:02:17.113531  2113 tablet_copy_client.cc:523] T 9e7759be006b4a8186aca434c48949b2 P eb5a2343514740caa85ff0228a535123: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:02:17.114334  2125 tablet_bootstrap.cc:492] T f55670daadbc4a7f88971471629c793f P eb5a2343514740caa85ff0228a535123: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:02:17.114766  2125 tablet_bootstrap.cc:492] T f55670daadbc4a7f88971471629c793f P eb5a2343514740caa85ff0228a535123: Bootstrap complete.
I20240502 02:02:17.115011  2125 ts_tablet_manager.cc:1367] T f55670daadbc4a7f88971471629c793f P eb5a2343514740caa85ff0228a535123: Time spent bootstrapping tablet: real 0.034s	user 0.032s	sys 0.000s
I20240502 02:02:17.115710  2125 raft_consensus.cc:348] T f55670daadbc4a7f88971471629c793f P eb5a2343514740caa85ff0228a535123 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } }
I20240502 02:02:17.115980  2125 raft_consensus.cc:729] T f55670daadbc4a7f88971471629c793f P eb5a2343514740caa85ff0228a535123 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb5a2343514740caa85ff0228a535123, State: Initialized, Role: LEARNER
I20240502 02:02:17.116266  2113 tablet_bootstrap.cc:492] T 9e7759be006b4a8186aca434c48949b2 P eb5a2343514740caa85ff0228a535123: Bootstrap starting.
I20240502 02:02:17.116191  2125 consensus_queue.cc:260] T f55670daadbc4a7f88971471629c793f P eb5a2343514740caa85ff0228a535123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } }
I20240502 02:02:17.116773  2125 ts_tablet_manager.cc:1398] T f55670daadbc4a7f88971471629c793f P eb5a2343514740caa85ff0228a535123: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240502 02:02:17.118283  1040 tablet_copy_service.cc:342] P d0e4135c07ac4f2389ffe4e7379df71f: Request end of tablet copy session eb5a2343514740caa85ff0228a535123-f55670daadbc4a7f88971471629c793f received from {username='slave'} at 127.25.109.132:40517
I20240502 02:02:17.118444  1040 tablet_copy_service.cc:433] P d0e4135c07ac4f2389ffe4e7379df71f: ending tablet copy session eb5a2343514740caa85ff0228a535123-f55670daadbc4a7f88971471629c793f on tablet f55670daadbc4a7f88971471629c793f with peer eb5a2343514740caa85ff0228a535123
I20240502 02:02:17.149520  2113 tablet_bootstrap.cc:492] T 9e7759be006b4a8186aca434c48949b2 P eb5a2343514740caa85ff0228a535123: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:02:17.149989  2113 tablet_bootstrap.cc:492] T 9e7759be006b4a8186aca434c48949b2 P eb5a2343514740caa85ff0228a535123: Bootstrap complete.
I20240502 02:02:17.150247  2113 ts_tablet_manager.cc:1367] T 9e7759be006b4a8186aca434c48949b2 P eb5a2343514740caa85ff0228a535123: Time spent bootstrapping tablet: real 0.034s	user 0.026s	sys 0.004s
I20240502 02:02:17.150926  2113 raft_consensus.cc:348] T 9e7759be006b4a8186aca434c48949b2 P eb5a2343514740caa85ff0228a535123 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { replace: true } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } }
I20240502 02:02:17.151121  2113 raft_consensus.cc:729] T 9e7759be006b4a8186aca434c48949b2 P eb5a2343514740caa85ff0228a535123 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb5a2343514740caa85ff0228a535123, State: Initialized, Role: LEARNER
I20240502 02:02:17.151315  2113 consensus_queue.cc:260] T 9e7759be006b4a8186aca434c48949b2 P eb5a2343514740caa85ff0228a535123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { replace: true } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } }
I20240502 02:02:17.151852  2113 ts_tablet_manager.cc:1398] T 9e7759be006b4a8186aca434c48949b2 P eb5a2343514740caa85ff0228a535123: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 02:02:17.152815  1176 tablet_copy_service.cc:342] P 8b9e878e72f64482bdd89f28ac427494: Request end of tablet copy session eb5a2343514740caa85ff0228a535123-9e7759be006b4a8186aca434c48949b2 received from {username='slave'} at 127.25.109.132:52325
I20240502 02:02:17.152977  1176 tablet_copy_service.cc:433] P 8b9e878e72f64482bdd89f28ac427494: ending tablet copy session eb5a2343514740caa85ff0228a535123-9e7759be006b4a8186aca434c48949b2 on tablet 9e7759be006b4a8186aca434c48949b2 with peer eb5a2343514740caa85ff0228a535123
I20240502 02:02:17.338848  1824 raft_consensus.cc:1203] T b1005c925b954ca39a9ee6f2677b3a1f P eb5a2343514740caa85ff0228a535123 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 02:02:17.338804  1825 raft_consensus.cc:1203] T b80d547302ec453aaf80fa6b64eebc13 P eb5a2343514740caa85ff0228a535123 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 02:02:17.514462  1825 raft_consensus.cc:1203] T 53d5bc373eb14d44bf1dd07375f40f00 P eb5a2343514740caa85ff0228a535123 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 02:02:17.523746  1825 raft_consensus.cc:1203] T f55670daadbc4a7f88971471629c793f P eb5a2343514740caa85ff0228a535123 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 02:02:17.725607  1825 raft_consensus.cc:1203] T 9e7759be006b4a8186aca434c48949b2 P eb5a2343514740caa85ff0228a535123 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 02:02:17.813967  2045 raft_consensus.cc:1050] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494: attempting to promote NON_VOTER eb5a2343514740caa85ff0228a535123 to VOTER
I20240502 02:02:17.815142  2045 consensus_queue.cc:237] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } }
I20240502 02:02:17.817869  1825 raft_consensus.cc:1261] T b80d547302ec453aaf80fa6b64eebc13 P eb5a2343514740caa85ff0228a535123 [term 1 LEARNER]: Refusing update from remote peer 8b9e878e72f64482bdd89f28ac427494: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:02:17.818728  2198 consensus_queue.cc:1035] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:17.818951  1018 raft_consensus.cc:1261] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Refusing update from remote peer 8b9e878e72f64482bdd89f28ac427494: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:02:17.819211  2197 raft_consensus.cc:1050] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25: attempting to promote NON_VOTER eb5a2343514740caa85ff0228a535123 to VOTER
I20240502 02:02:17.819609  1288 raft_consensus.cc:1261] T b80d547302ec453aaf80fa6b64eebc13 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Refusing update from remote peer 8b9e878e72f64482bdd89f28ac427494: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:02:17.820255  2198 consensus_queue.cc:1035] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:17.820358  2197 consensus_queue.cc:237] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { replace: true } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } }
I20240502 02:02:17.820613  1658 consensus_queue.cc:1035] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:17.824183  1824 raft_consensus.cc:1261] T b1005c925b954ca39a9ee6f2677b3a1f P eb5a2343514740caa85ff0228a535123 [term 1 LEARNER]: Refusing update from remote peer d37e50431f30463c872c5d819ddf2c25: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:02:17.824364  1156 raft_consensus.cc:1261] T b1005c925b954ca39a9ee6f2677b3a1f P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Refusing update from remote peer d37e50431f30463c872c5d819ddf2c25: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:02:17.824853  1731 consensus_queue.cc:1035] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:17.825215  2196 consensus_queue.cc:1035] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:17.825937  2198 raft_consensus.cc:2932] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, eb5a2343514740caa85ff0228a535123 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:17.829639  1018 raft_consensus.cc:1261] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Refusing update from remote peer d37e50431f30463c872c5d819ddf2c25: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:02:17.829632  2196 raft_consensus.cc:2932] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, eb5a2343514740caa85ff0228a535123 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { replace: true } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:17.830339  1824 raft_consensus.cc:2932] T b1005c925b954ca39a9ee6f2677b3a1f P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, eb5a2343514740caa85ff0228a535123 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { replace: true } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:17.831847  1018 raft_consensus.cc:2932] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, eb5a2343514740caa85ff0228a535123 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:17.833396  1731 consensus_queue.cc:1035] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:17.834789   866 catalog_manager.cc:5484] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 reported cstate change: config changed from index 4 to 5, eb5a2343514740caa85ff0228a535123 (127.25.109.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8b9e878e72f64482bdd89f28ac427494" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 02:02:17.835323  1019 raft_consensus.cc:2932] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, eb5a2343514740caa85ff0228a535123 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { replace: true } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:17.837915  1825 raft_consensus.cc:2932] T b80d547302ec453aaf80fa6b64eebc13 P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, eb5a2343514740caa85ff0228a535123 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:17.843520  1288 raft_consensus.cc:2932] T b80d547302ec453aaf80fa6b64eebc13 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, eb5a2343514740caa85ff0228a535123 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:17.844779   867 catalog_manager.cc:5484] T b1005c925b954ca39a9ee6f2677b3a1f P eb5a2343514740caa85ff0228a535123 reported cstate change: config changed from index 4 to 5, eb5a2343514740caa85ff0228a535123 (127.25.109.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d37e50431f30463c872c5d819ddf2c25" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { replace: true } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:17.861584  1156 raft_consensus.cc:2932] T b1005c925b954ca39a9ee6f2677b3a1f P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, eb5a2343514740caa85ff0228a535123 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { replace: true } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:17.869426  1288 consensus_queue.cc:237] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } }
I20240502 02:02:17.871405  1156 raft_consensus.cc:1261] T b1005c925b954ca39a9ee6f2677b3a1f P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Refusing update from remote peer d37e50431f30463c872c5d819ddf2c25: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:02:17.871503  1825 raft_consensus.cc:1261] T b1005c925b954ca39a9ee6f2677b3a1f P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Refusing update from remote peer d37e50431f30463c872c5d819ddf2c25: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:02:17.872213  2256 consensus_queue.cc:1035] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:02:17.872596  2197 consensus_queue.cc:1035] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:02:17.875165  2196 raft_consensus.cc:2932] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:17.875833  1156 raft_consensus.cc:2932] T b1005c925b954ca39a9ee6f2677b3a1f P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:17.875900  1825 raft_consensus.cc:2932] T b1005c925b954ca39a9ee6f2677b3a1f P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:17.879432   859 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b1005c925b954ca39a9ee6f2677b3a1f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 02:02:17.880162   873 catalog_manager.cc:5484] T b1005c925b954ca39a9ee6f2677b3a1f P 8b9e878e72f64482bdd89f28ac427494 reported cstate change: config changed from index 5 to 6, VOTER d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129) evicted. New cstate: current_term: 1 leader_uuid: "d37e50431f30463c872c5d819ddf2c25" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:17.884721   998 tablet_service.cc:1514] Processing DeleteTablet for tablet b1005c925b954ca39a9ee6f2677b3a1f with delete_type TABLET_DATA_TOMBSTONED (TS d0e4135c07ac4f2389ffe4e7379df71f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33552
I20240502 02:02:17.885601  2269 tablet_replica.cc:330] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f: stopping tablet replica
I20240502 02:02:17.885994  2269 raft_consensus.cc:2229] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 02:02:17.886281  2269 raft_consensus.cc:2258] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 02:02:17.887665  2269 ts_tablet_manager.cc:1875] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 02:02:17.892755  2269 ts_tablet_manager.cc:1888] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 02:02:17.892904  2269 log.cc:1196] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal/wals/b1005c925b954ca39a9ee6f2677b3a1f
I20240502 02:02:17.893898   860 catalog_manager.cc:4830] TS d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129:33963): tablet b1005c925b954ca39a9ee6f2677b3a1f (table rebalance_test_table_1 [id=8991230805234f99bb3820fc5a537d77]) successfully deleted
I20240502 02:02:17.974572  2200 raft_consensus.cc:1050] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f: attempting to promote NON_VOTER eb5a2343514740caa85ff0228a535123 to VOTER
I20240502 02:02:17.975474  2200 consensus_queue.cc:237] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } }
I20240502 02:02:17.978263  1156 raft_consensus.cc:1261] T 53d5bc373eb14d44bf1dd07375f40f00 P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:02:17.978274  1825 raft_consensus.cc:1261] T 53d5bc373eb14d44bf1dd07375f40f00 P eb5a2343514740caa85ff0228a535123 [term 1 LEARNER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:02:17.978264  1288 raft_consensus.cc:1261] T 53d5bc373eb14d44bf1dd07375f40f00 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:02:17.978945  2200 consensus_queue.cc:1035] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:17.979307  2014 consensus_queue.cc:1035] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:17.979645  2271 consensus_queue.cc:1035] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:17.983464  2014 raft_consensus.cc:2932] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, eb5a2343514740caa85ff0228a535123 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:17.984670  1825 raft_consensus.cc:2932] T 53d5bc373eb14d44bf1dd07375f40f00 P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, eb5a2343514740caa85ff0228a535123 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:17.987083  1156 raft_consensus.cc:2932] T 53d5bc373eb14d44bf1dd07375f40f00 P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, eb5a2343514740caa85ff0228a535123 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:17.988281  1288 raft_consensus.cc:2932] T 53d5bc373eb14d44bf1dd07375f40f00 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, eb5a2343514740caa85ff0228a535123 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:17.990166   867 catalog_manager.cc:5484] T 53d5bc373eb14d44bf1dd07375f40f00 P eb5a2343514740caa85ff0228a535123 reported cstate change: config changed from index 4 to 5, eb5a2343514740caa85ff0228a535123 (127.25.109.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:17.996104  1019 consensus_queue.cc:237] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } }
I20240502 02:02:17.998037  1288 raft_consensus.cc:1261] T 53d5bc373eb14d44bf1dd07375f40f00 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:02:17.998569  2200 consensus_queue.cc:1035] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:02:17.998642  1825 raft_consensus.cc:1261] T 53d5bc373eb14d44bf1dd07375f40f00 P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:02:17.999148  2271 consensus_queue.cc:1035] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:02:18.000945  2200 raft_consensus.cc:2932] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:18.001618  1288 raft_consensus.cc:2932] T 53d5bc373eb14d44bf1dd07375f40f00 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:18.002310  1825 raft_consensus.cc:2932] T 53d5bc373eb14d44bf1dd07375f40f00 P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:18.004138   860 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 53d5bc373eb14d44bf1dd07375f40f00 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 02:02:18.005694   866 catalog_manager.cc:5484] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f reported cstate change: config changed from index 5 to 6, VOTER 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130) evicted. New cstate: current_term: 1 leader_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 02:02:18.009843  1135 tablet_service.cc:1514] Processing DeleteTablet for tablet 53d5bc373eb14d44bf1dd07375f40f00 with delete_type TABLET_DATA_TOMBSTONED (TS 8b9e878e72f64482bdd89f28ac427494 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:32784
I20240502 02:02:18.012012  2275 tablet_replica.cc:330] T 53d5bc373eb14d44bf1dd07375f40f00 P 8b9e878e72f64482bdd89f28ac427494: stopping tablet replica
I20240502 02:02:18.012579  2275 raft_consensus.cc:2229] T 53d5bc373eb14d44bf1dd07375f40f00 P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 02:02:18.013073  2275 raft_consensus.cc:2258] T 53d5bc373eb14d44bf1dd07375f40f00 P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 02:02:18.014609  2275 ts_tablet_manager.cc:1875] T 53d5bc373eb14d44bf1dd07375f40f00 P 8b9e878e72f64482bdd89f28ac427494: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 02:02:18.018927  2199 raft_consensus.cc:1050] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f: attempting to promote NON_VOTER eb5a2343514740caa85ff0228a535123 to VOTER
I20240502 02:02:18.019701  2199 consensus_queue.cc:237] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } }
I20240502 02:02:18.019999  2275 ts_tablet_manager.cc:1888] T 53d5bc373eb14d44bf1dd07375f40f00 P 8b9e878e72f64482bdd89f28ac427494: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 02:02:18.020195  2275 log.cc:1196] T 53d5bc373eb14d44bf1dd07375f40f00 P 8b9e878e72f64482bdd89f28ac427494: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal/wals/53d5bc373eb14d44bf1dd07375f40f00
I20240502 02:02:18.021160   861 catalog_manager.cc:4830] TS 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130:43631): tablet 53d5bc373eb14d44bf1dd07375f40f00 (table rebalance_test_table_2 [id=57c9d0a8579e4e908683ee50f24d5aa1]) successfully deleted
I20240502 02:02:18.022468  1156 raft_consensus.cc:1261] T f55670daadbc4a7f88971471629c793f P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:02:18.022521  1825 raft_consensus.cc:1261] T f55670daadbc4a7f88971471629c793f P eb5a2343514740caa85ff0228a535123 [term 1 LEARNER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:02:18.023063  1288 raft_consensus.cc:1261] T f55670daadbc4a7f88971471629c793f P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:02:18.023133  2014 consensus_queue.cc:1035] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:18.023834  2271 consensus_queue.cc:1035] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:18.024225  2200 consensus_queue.cc:1035] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:18.028301  2014 raft_consensus.cc:2932] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, eb5a2343514740caa85ff0228a535123 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:18.029161  1825 raft_consensus.cc:2932] T f55670daadbc4a7f88971471629c793f P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, eb5a2343514740caa85ff0228a535123 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:18.034672   866 catalog_manager.cc:5484] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f reported cstate change: config changed from index 4 to 5, eb5a2343514740caa85ff0228a535123 (127.25.109.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 02:02:18.035709  1156 raft_consensus.cc:2932] T f55670daadbc4a7f88971471629c793f P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, eb5a2343514740caa85ff0228a535123 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:18.036475  1288 raft_consensus.cc:2932] T f55670daadbc4a7f88971471629c793f P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, eb5a2343514740caa85ff0228a535123 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:18.040472  1019 consensus_queue.cc:237] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } }
I20240502 02:02:18.042862  1825 raft_consensus.cc:1261] T f55670daadbc4a7f88971471629c793f P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:02:18.043516  2199 consensus_queue.cc:1035] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:02:18.043943  1288 raft_consensus.cc:1261] T f55670daadbc4a7f88971471629c793f P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:02:18.044546  2199 consensus_queue.cc:1035] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:02:18.047093  2199 raft_consensus.cc:2932] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:18.049095  1288 raft_consensus.cc:2932] T f55670daadbc4a7f88971471629c793f P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:18.051290   860 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f55670daadbc4a7f88971471629c793f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 02:02:18.051823  1825 raft_consensus.cc:2932] T f55670daadbc4a7f88971471629c793f P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:18.052672   867 catalog_manager.cc:5484] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f reported cstate change: config changed from index 5 to 6, VOTER 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130) evicted. New cstate: current_term: 1 leader_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 02:02:18.058305  1135 tablet_service.cc:1514] Processing DeleteTablet for tablet f55670daadbc4a7f88971471629c793f with delete_type TABLET_DATA_TOMBSTONED (TS 8b9e878e72f64482bdd89f28ac427494 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:32784
I20240502 02:02:18.058727  2275 tablet_replica.cc:330] T f55670daadbc4a7f88971471629c793f P 8b9e878e72f64482bdd89f28ac427494: stopping tablet replica
I20240502 02:02:18.058936  2275 raft_consensus.cc:2229] T f55670daadbc4a7f88971471629c793f P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 02:02:18.059190  2275 raft_consensus.cc:2258] T f55670daadbc4a7f88971471629c793f P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 02:02:18.060225  2275 ts_tablet_manager.cc:1875] T f55670daadbc4a7f88971471629c793f P 8b9e878e72f64482bdd89f28ac427494: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 02:02:18.065563  2275 ts_tablet_manager.cc:1888] T f55670daadbc4a7f88971471629c793f P 8b9e878e72f64482bdd89f28ac427494: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 02:02:18.065788  2275 log.cc:1196] T f55670daadbc4a7f88971471629c793f P 8b9e878e72f64482bdd89f28ac427494: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal/wals/f55670daadbc4a7f88971471629c793f
I20240502 02:02:18.066609   861 catalog_manager.cc:4830] TS 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130:43631): tablet f55670daadbc4a7f88971471629c793f (table rebalance_test_table_1 [id=8991230805234f99bb3820fc5a537d77]) successfully deleted
I20240502 02:02:18.090274  1156 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b80d547302ec453aaf80fa6b64eebc13"
dest_uuid: "8b9e878e72f64482bdd89f28ac427494"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:40326
I20240502 02:02:18.090566  1156 raft_consensus.cc:595] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [term 1 LEADER]: Received request to transfer leadership
I20240502 02:02:18.105206  2045 raft_consensus.cc:1050] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494: attempting to promote NON_VOTER eb5a2343514740caa85ff0228a535123 to VOTER
I20240502 02:02:18.106268  2045 consensus_queue.cc:237] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { replace: true } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } }
I20240502 02:02:18.108968  1288 raft_consensus.cc:1261] T 9e7759be006b4a8186aca434c48949b2 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Refusing update from remote peer 8b9e878e72f64482bdd89f28ac427494: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:02:18.109390  1825 raft_consensus.cc:1261] T 9e7759be006b4a8186aca434c48949b2 P eb5a2343514740caa85ff0228a535123 [term 1 LEARNER]: Refusing update from remote peer 8b9e878e72f64482bdd89f28ac427494: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:02:18.109401  1019 raft_consensus.cc:1261] T 9e7759be006b4a8186aca434c48949b2 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Refusing update from remote peer 8b9e878e72f64482bdd89f28ac427494: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:02:18.109634  2198 consensus_queue.cc:1035] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:18.109958  2045 consensus_queue.cc:1035] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:18.110600  2045 consensus_queue.cc:1035] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:18.114235  2198 raft_consensus.cc:2932] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, eb5a2343514740caa85ff0228a535123 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { replace: true } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:18.114866  1825 raft_consensus.cc:2932] T 9e7759be006b4a8186aca434c48949b2 P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, eb5a2343514740caa85ff0228a535123 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { replace: true } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:18.115170  1019 raft_consensus.cc:2932] T 9e7759be006b4a8186aca434c48949b2 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, eb5a2343514740caa85ff0228a535123 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { replace: true } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:18.119463  1288 raft_consensus.cc:2932] T 9e7759be006b4a8186aca434c48949b2 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, eb5a2343514740caa85ff0228a535123 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { replace: true } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:18.120887   867 catalog_manager.cc:5484] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 reported cstate change: config changed from index 4 to 5, eb5a2343514740caa85ff0228a535123 (127.25.109.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8b9e878e72f64482bdd89f28ac427494" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 02:02:18.126579  1156 consensus_queue.cc:237] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } }
I20240502 02:02:18.128471  1825 raft_consensus.cc:1261] T 9e7759be006b4a8186aca434c48949b2 P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Refusing update from remote peer 8b9e878e72f64482bdd89f28ac427494: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:02:18.128543  1288 raft_consensus.cc:1261] T 9e7759be006b4a8186aca434c48949b2 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Refusing update from remote peer 8b9e878e72f64482bdd89f28ac427494: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:02:18.129045  2264 consensus_queue.cc:1035] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:02:18.129374  2045 consensus_queue.cc:1035] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:02:18.132017  1658 raft_consensus.cc:2932] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:18.132547  1288 raft_consensus.cc:2932] T 9e7759be006b4a8186aca434c48949b2 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:18.132557  1825 raft_consensus.cc:2932] T 9e7759be006b4a8186aca434c48949b2 P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:18.138646   868 catalog_manager.cc:5484] T 9e7759be006b4a8186aca434c48949b2 P eb5a2343514740caa85ff0228a535123 reported cstate change: config changed from index 5 to 6, VOTER d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129) evicted. New cstate: current_term: 1 leader_uuid: "8b9e878e72f64482bdd89f28ac427494" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:18.140780   861 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9e7759be006b4a8186aca434c48949b2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 02:02:18.143697   998 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e7759be006b4a8186aca434c48949b2 with delete_type TABLET_DATA_TOMBSTONED (TS d0e4135c07ac4f2389ffe4e7379df71f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33552
I20240502 02:02:18.144243  2269 tablet_replica.cc:330] T 9e7759be006b4a8186aca434c48949b2 P d0e4135c07ac4f2389ffe4e7379df71f: stopping tablet replica
I20240502 02:02:18.144475  2269 raft_consensus.cc:2229] T 9e7759be006b4a8186aca434c48949b2 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 02:02:18.144896  2269 raft_consensus.cc:2258] T 9e7759be006b4a8186aca434c48949b2 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 02:02:18.146101  2269 ts_tablet_manager.cc:1875] T 9e7759be006b4a8186aca434c48949b2 P d0e4135c07ac4f2389ffe4e7379df71f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 02:02:18.149008  1288 consensus_queue.cc:237] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { replace: true } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } }
I20240502 02:02:18.150801  2269 ts_tablet_manager.cc:1888] T 9e7759be006b4a8186aca434c48949b2 P d0e4135c07ac4f2389ffe4e7379df71f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 02:02:18.151019  2269 log.cc:1196] T 9e7759be006b4a8186aca434c48949b2 P d0e4135c07ac4f2389ffe4e7379df71f: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal/wals/9e7759be006b4a8186aca434c48949b2
I20240502 02:02:18.151427  1156 raft_consensus.cc:1261] T 884b0248d66f4575a7e8e3ead42c58c6 P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Refusing update from remote peer d37e50431f30463c872c5d819ddf2c25: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 02:02:18.151680  1019 raft_consensus.cc:1261] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Refusing update from remote peer d37e50431f30463c872c5d819ddf2c25: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 02:02:18.152014   860 catalog_manager.cc:4830] TS d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129:33963): tablet 9e7759be006b4a8186aca434c48949b2 (table rebalance_test_table_0 [id=3920dc53a6a846338bdd42178520e7af]) successfully deleted
I20240502 02:02:18.152164  2268 consensus_queue.cc:1035] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:02:18.152487  2197 consensus_queue.cc:1035] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 02:02:18.152786  1223 consensus_peers.cc:485] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 -> Peer eb5a2343514740caa85ff0228a535123 (127.25.109.132:44441): Couldn't send request to peer eb5a2343514740caa85ff0228a535123. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 884b0248d66f4575a7e8e3ead42c58c6. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:02:18.155279  2256 raft_consensus.cc:2932] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER eb5a2343514740caa85ff0228a535123 (127.25.109.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { replace: true } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } } }
I20240502 02:02:18.155995  1019 raft_consensus.cc:2932] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER eb5a2343514740caa85ff0228a535123 (127.25.109.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { replace: true } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } } }
I20240502 02:02:18.155977  1156 raft_consensus.cc:2932] T 884b0248d66f4575a7e8e3ead42c58c6 P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER eb5a2343514740caa85ff0228a535123 (127.25.109.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { replace: true } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } } }
I20240502 02:02:18.160637   866 catalog_manager.cc:5484] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f reported cstate change: config changed from index -1 to 4, NON_VOTER eb5a2343514740caa85ff0228a535123 (127.25.109.132) added. New cstate: current_term: 1 leader_uuid: "d37e50431f30463c872c5d819ddf2c25" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { replace: true } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } } }
I20240502 02:02:18.212500  1156 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b80d547302ec453aaf80fa6b64eebc13"
dest_uuid: "8b9e878e72f64482bdd89f28ac427494"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:40348
I20240502 02:02:18.212751  1156 raft_consensus.cc:595] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [term 1 LEADER]: Received request to transfer leadership
I20240502 02:02:18.240192  1156 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b80d547302ec453aaf80fa6b64eebc13"
dest_uuid: "8b9e878e72f64482bdd89f28ac427494"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:40370
I20240502 02:02:18.240430  1156 raft_consensus.cc:595] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [term 1 LEADER]: Received request to transfer leadership
I20240502 02:02:18.329885  2198 raft_consensus.cc:979] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494: : Instructing follower d0e4135c07ac4f2389ffe4e7379df71f to start an election
I20240502 02:02:18.330096  2198 raft_consensus.cc:1067] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [term 1 LEADER]: Signalling peer d0e4135c07ac4f2389ffe4e7379df71f to start an election
I20240502 02:02:18.331010  1019 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b80d547302ec453aaf80fa6b64eebc13"
dest_uuid: "d0e4135c07ac4f2389ffe4e7379df71f"
 from {username='slave'} at 127.25.109.130:37059
I20240502 02:02:18.331311  1019 raft_consensus.cc:482] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 02:02:18.331429  1019 raft_consensus.cc:3037] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Advancing to term 2
I20240502 02:02:18.333618  1019 raft_consensus.cc:504] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } }
I20240502 02:02:18.334481  1019 leader_election.cc:288] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f [CANDIDATE]: Term 2 election: Requested vote from peers 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130:43631), d37e50431f30463c872c5d819ddf2c25 (127.25.109.131:37799), eb5a2343514740caa85ff0228a535123 (127.25.109.132:44441)
I20240502 02:02:18.334961  1156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b80d547302ec453aaf80fa6b64eebc13" candidate_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "8b9e878e72f64482bdd89f28ac427494"
I20240502 02:02:18.335013  1288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b80d547302ec453aaf80fa6b64eebc13" candidate_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "d37e50431f30463c872c5d819ddf2c25"
I20240502 02:02:18.335238  1288 raft_consensus.cc:3037] T b80d547302ec453aaf80fa6b64eebc13 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Advancing to term 2
I20240502 02:02:18.335255  1156 raft_consensus.cc:3032] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 02:02:18.335345  1156 raft_consensus.cc:729] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8b9e878e72f64482bdd89f28ac427494, State: Running, Role: LEADER
I20240502 02:02:18.335326  1825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b80d547302ec453aaf80fa6b64eebc13" candidate_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "eb5a2343514740caa85ff0228a535123"
I20240502 02:02:18.335673  1825 raft_consensus.cc:3037] T b80d547302ec453aaf80fa6b64eebc13 P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Advancing to term 2
I20240502 02:02:18.335686  1156 consensus_queue.cc:260] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } }
I20240502 02:02:18.336233  1156 raft_consensus.cc:3037] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Advancing to term 2
I20240502 02:02:18.338112  1825 raft_consensus.cc:2454] T b80d547302ec453aaf80fa6b64eebc13 P eb5a2343514740caa85ff0228a535123 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0e4135c07ac4f2389ffe4e7379df71f in term 2.
I20240502 02:02:18.338215  1288 raft_consensus.cc:2454] T b80d547302ec453aaf80fa6b64eebc13 P d37e50431f30463c872c5d819ddf2c25 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0e4135c07ac4f2389ffe4e7379df71f in term 2.
I20240502 02:02:18.338791   951 leader_election.cc:302] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: d0e4135c07ac4f2389ffe4e7379df71f, d37e50431f30463c872c5d819ddf2c25, eb5a2343514740caa85ff0228a535123; no voters: 
I20240502 02:02:18.339004  1156 raft_consensus.cc:2454] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0e4135c07ac4f2389ffe4e7379df71f in term 2.
I20240502 02:02:18.339143  2199 raft_consensus.cc:2781] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 02:02:18.339341  2199 raft_consensus.cc:686] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f [term 2 LEADER]: Becoming Leader. State: Replica: d0e4135c07ac4f2389ffe4e7379df71f, State: Running, Role: LEADER
I20240502 02:02:18.339676  2199 consensus_queue.cc:237] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } }
I20240502 02:02:18.342675   867 catalog_manager.cc:5484] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f reported cstate change: term changed from 1 to 2, leader changed from 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130) to d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129). New cstate: current_term: 2 leader_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 02:02:18.749111  1156 raft_consensus.cc:1261] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [term 2 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 02:02:18.749748  2199 consensus_queue.cc:1035] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:02:18.758651  2386 ts_tablet_manager.cc:902] T 884b0248d66f4575a7e8e3ead42c58c6 P eb5a2343514740caa85ff0228a535123: Initiating tablet copy from peer d37e50431f30463c872c5d819ddf2c25 (127.25.109.131:37799)
I20240502 02:02:18.760241  2386 tablet_copy_client.cc:308] T 884b0248d66f4575a7e8e3ead42c58c6 P eb5a2343514740caa85ff0228a535123: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.131:37799
I20240502 02:02:18.760416  1825 raft_consensus.cc:1261] T b80d547302ec453aaf80fa6b64eebc13 P eb5a2343514740caa85ff0228a535123 [term 2 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 02:02:18.761009  1309 tablet_copy_service.cc:140] P d37e50431f30463c872c5d819ddf2c25: Received BeginTabletCopySession request for tablet 884b0248d66f4575a7e8e3ead42c58c6 from peer eb5a2343514740caa85ff0228a535123 ({username='slave'} at 127.25.109.132:43379)
I20240502 02:02:18.761018  2199 consensus_queue.cc:1035] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:02:18.761194  1309 tablet_copy_service.cc:161] P d37e50431f30463c872c5d819ddf2c25: Beginning new tablet copy session on tablet 884b0248d66f4575a7e8e3ead42c58c6 from peer eb5a2343514740caa85ff0228a535123 at {username='slave'} at 127.25.109.132:43379: session id = eb5a2343514740caa85ff0228a535123-884b0248d66f4575a7e8e3ead42c58c6
I20240502 02:02:18.763726  1309 tablet_copy_source_session.cc:205] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 02:02:18.765889  1288 raft_consensus.cc:1261] T b80d547302ec453aaf80fa6b64eebc13 P d37e50431f30463c872c5d819ddf2c25 [term 2 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 02:02:18.766546  2386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 884b0248d66f4575a7e8e3ead42c58c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:18.766592  2199 consensus_queue.cc:1035] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:02:18.777611  2386 tablet_copy_client.cc:783] T 884b0248d66f4575a7e8e3ead42c58c6 P eb5a2343514740caa85ff0228a535123: tablet copy: Starting download of 0 data blocks...
I20240502 02:02:18.778090  2386 tablet_copy_client.cc:647] T 884b0248d66f4575a7e8e3ead42c58c6 P eb5a2343514740caa85ff0228a535123: tablet copy: Starting download of 1 WAL segments...
I20240502 02:02:18.780515  2386 tablet_copy_client.cc:523] T 884b0248d66f4575a7e8e3ead42c58c6 P eb5a2343514740caa85ff0228a535123: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:02:18.784642  2386 tablet_bootstrap.cc:492] T 884b0248d66f4575a7e8e3ead42c58c6 P eb5a2343514740caa85ff0228a535123: Bootstrap starting.
I20240502 02:02:18.815614  2386 tablet_bootstrap.cc:492] T 884b0248d66f4575a7e8e3ead42c58c6 P eb5a2343514740caa85ff0228a535123: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:02:18.816068  2386 tablet_bootstrap.cc:492] T 884b0248d66f4575a7e8e3ead42c58c6 P eb5a2343514740caa85ff0228a535123: Bootstrap complete.
I20240502 02:02:18.816391  2386 ts_tablet_manager.cc:1367] T 884b0248d66f4575a7e8e3ead42c58c6 P eb5a2343514740caa85ff0228a535123: Time spent bootstrapping tablet: real 0.032s	user 0.028s	sys 0.004s
I20240502 02:02:18.817153  2386 raft_consensus.cc:348] T 884b0248d66f4575a7e8e3ead42c58c6 P eb5a2343514740caa85ff0228a535123 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { replace: true } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } }
I20240502 02:02:18.817399  2386 raft_consensus.cc:729] T 884b0248d66f4575a7e8e3ead42c58c6 P eb5a2343514740caa85ff0228a535123 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb5a2343514740caa85ff0228a535123, State: Initialized, Role: LEARNER
I20240502 02:02:18.817606  2386 consensus_queue.cc:260] T 884b0248d66f4575a7e8e3ead42c58c6 P eb5a2343514740caa85ff0228a535123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { replace: true } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } }
I20240502 02:02:18.818105  2386 ts_tablet_manager.cc:1398] T 884b0248d66f4575a7e8e3ead42c58c6 P eb5a2343514740caa85ff0228a535123: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 02:02:18.819020  1309 tablet_copy_service.cc:342] P d37e50431f30463c872c5d819ddf2c25: Request end of tablet copy session eb5a2343514740caa85ff0228a535123-884b0248d66f4575a7e8e3ead42c58c6 received from {username='slave'} at 127.25.109.132:43379
I20240502 02:02:18.819190  1309 tablet_copy_service.cc:433] P d37e50431f30463c872c5d819ddf2c25: ending tablet copy session eb5a2343514740caa85ff0228a535123-884b0248d66f4575a7e8e3ead42c58c6 on tablet 884b0248d66f4575a7e8e3ead42c58c6 with peer eb5a2343514740caa85ff0228a535123
I20240502 02:02:19.191073  1019 consensus_queue.cc:237] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } }
I20240502 02:02:19.192955  1825 raft_consensus.cc:1261] T b80d547302ec453aaf80fa6b64eebc13 P eb5a2343514740caa85ff0228a535123 [term 2 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 02:02:19.193296  1288 raft_consensus.cc:1261] T b80d547302ec453aaf80fa6b64eebc13 P d37e50431f30463c872c5d819ddf2c25 [term 2 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 02:02:19.193555  2200 consensus_queue.cc:1035] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 02:02:19.193884  2394 consensus_queue.cc:1035] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 02:02:19.196127  2199 raft_consensus.cc:2932] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:19.196852  1288 raft_consensus.cc:2932] T b80d547302ec453aaf80fa6b64eebc13 P d37e50431f30463c872c5d819ddf2c25 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:19.196823  1825 raft_consensus.cc:2932] T b80d547302ec453aaf80fa6b64eebc13 P eb5a2343514740caa85ff0228a535123 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:19.200030   860 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b80d547302ec453aaf80fa6b64eebc13 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 02:02:19.201284   873 catalog_manager.cc:5484] T b80d547302ec453aaf80fa6b64eebc13 P d37e50431f30463c872c5d819ddf2c25 reported cstate change: config changed from index 5 to 7, VOTER 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130) evicted. New cstate: current_term: 2 leader_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:19.211776  1135 tablet_service.cc:1514] Processing DeleteTablet for tablet b80d547302ec453aaf80fa6b64eebc13 with delete_type TABLET_DATA_TOMBSTONED (TS 8b9e878e72f64482bdd89f28ac427494 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:32784
I20240502 02:02:19.212426  2424 tablet_replica.cc:330] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494: stopping tablet replica
I20240502 02:02:19.212605  2424 raft_consensus.cc:2229] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 02:02:19.212774  2424 raft_consensus.cc:2258] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 02:02:19.213483  2424 ts_tablet_manager.cc:1875] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 02:02:19.217936  2424 ts_tablet_manager.cc:1888] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 02:02:19.218050  2424 log.cc:1196] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-1/wal/wals/b80d547302ec453aaf80fa6b64eebc13
I20240502 02:02:19.218750   861 catalog_manager.cc:4830] TS 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130:43631): tablet b80d547302ec453aaf80fa6b64eebc13 (table rebalance_test_table_0 [id=3920dc53a6a846338bdd42178520e7af]) successfully deleted
I20240502 02:02:19.291427  1825 raft_consensus.cc:1203] T 884b0248d66f4575a7e8e3ead42c58c6 P eb5a2343514740caa85ff0228a535123 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 02:02:19.906633  2454 raft_consensus.cc:1050] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25: attempting to promote NON_VOTER eb5a2343514740caa85ff0228a535123 to VOTER
I20240502 02:02:19.907387  2454 consensus_queue.cc:237] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { replace: true } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } }
I20240502 02:02:19.909864  1825 raft_consensus.cc:1261] T 884b0248d66f4575a7e8e3ead42c58c6 P eb5a2343514740caa85ff0228a535123 [term 1 LEARNER]: Refusing update from remote peer d37e50431f30463c872c5d819ddf2c25: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:02:19.910005  1156 raft_consensus.cc:1261] T 884b0248d66f4575a7e8e3ead42c58c6 P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Refusing update from remote peer d37e50431f30463c872c5d819ddf2c25: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:02:19.910588  1019 raft_consensus.cc:1261] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Refusing update from remote peer d37e50431f30463c872c5d819ddf2c25: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:02:19.910642  2456 consensus_queue.cc:1035] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:19.911087  2454 consensus_queue.cc:1035] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:19.911473  2426 consensus_queue.cc:1035] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:19.914520  2456 raft_consensus.cc:2932] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, eb5a2343514740caa85ff0228a535123 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { replace: true } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:19.915220  1019 raft_consensus.cc:2932] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, eb5a2343514740caa85ff0228a535123 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { replace: true } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:19.916378  1156 raft_consensus.cc:2932] T 884b0248d66f4575a7e8e3ead42c58c6 P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, eb5a2343514740caa85ff0228a535123 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { replace: true } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:19.918653  1825 raft_consensus.cc:2932] T 884b0248d66f4575a7e8e3ead42c58c6 P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, eb5a2343514740caa85ff0228a535123 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { replace: true } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:19.919966   873 catalog_manager.cc:5484] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 reported cstate change: config changed from index 4 to 5, eb5a2343514740caa85ff0228a535123 (127.25.109.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d37e50431f30463c872c5d819ddf2c25" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 02:02:19.928362  1288 consensus_queue.cc:237] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } }
I20240502 02:02:19.930472  1825 raft_consensus.cc:1261] T 884b0248d66f4575a7e8e3ead42c58c6 P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Refusing update from remote peer d37e50431f30463c872c5d819ddf2c25: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:02:19.931146  2456 consensus_queue.cc:1035] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:02:19.933761  2454 raft_consensus.cc:2932] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:19.934451  1825 raft_consensus.cc:2932] T 884b0248d66f4575a7e8e3ead42c58c6 P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:19.936666   859 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 884b0248d66f4575a7e8e3ead42c58c6 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 02:02:19.939939   868 catalog_manager.cc:5484] T 884b0248d66f4575a7e8e3ead42c58c6 P eb5a2343514740caa85ff0228a535123 reported cstate change: config changed from index 5 to 6, VOTER d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129) evicted. New cstate: current_term: 1 leader_uuid: "d37e50431f30463c872c5d819ddf2c25" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:19.940294  1156 raft_consensus.cc:1261] T 884b0248d66f4575a7e8e3ead42c58c6 P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Refusing update from remote peer d37e50431f30463c872c5d819ddf2c25: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:02:19.940888  2456 consensus_queue.cc:1035] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:02:19.942582  1156 raft_consensus.cc:2932] T 884b0248d66f4575a7e8e3ead42c58c6 P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:19.944101   998 tablet_service.cc:1514] Processing DeleteTablet for tablet 884b0248d66f4575a7e8e3ead42c58c6 with delete_type TABLET_DATA_TOMBSTONED (TS d0e4135c07ac4f2389ffe4e7379df71f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33552
I20240502 02:02:19.952948  2464 tablet_replica.cc:330] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f: stopping tablet replica
I20240502 02:02:19.953563  2464 raft_consensus.cc:2229] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 02:02:19.953919  2464 raft_consensus.cc:2258] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 02:02:19.954777  2464 ts_tablet_manager.cc:1875] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 02:02:19.959223  2464 ts_tablet_manager.cc:1888] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 02:02:19.959338  2464 log.cc:1196] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-0/wal/wals/884b0248d66f4575a7e8e3ead42c58c6
I20240502 02:02:19.960078   860 catalog_manager.cc:4830] TS d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129:33963): tablet 884b0248d66f4575a7e8e3ead42c58c6 (table rebalance_test_table_2 [id=57c9d0a8579e4e908683ee50f24d5aa1]) successfully deleted
I20240502 02:02:20.077992  1135 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:20.091341  1269 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 02:02:20.091892   998 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:02:20.096179  1804 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:02:20.251292  1135 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:20.265314  1269 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 02:02:20.275009  1804 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:02:20.306908   998 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:02:20.448448   998 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:02:20.452204  1135 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:20.460319  1804 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:02:20.464696  1269 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 02:02:21.050699   866 ts_manager.cc:292] Set tserver state for d37e50431f30463c872c5d819ddf2c25 to MAINTENANCE_MODE
I20240502 02:02:22.507272  1135 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:22.514863   998 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:02:22.521401  1804 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:02:22.562938  1269 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 02:02:22.708622  1135 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:22.716722   998 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:02:22.717631  1269 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 02:02:22.723760  1804 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:02:22.894557  1135 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:22.896039  1269 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 02:02:22.898058   998 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 02:02:22.912650  1804 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:02:22.975651  1019 consensus_queue.cc:237] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } }
I20240502 02:02:22.977991  1825 raft_consensus.cc:1261] T b80d547302ec453aaf80fa6b64eebc13 P eb5a2343514740caa85ff0228a535123 [term 2 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 02:02:22.978034  1288 raft_consensus.cc:1261] T b80d547302ec453aaf80fa6b64eebc13 P d37e50431f30463c872c5d819ddf2c25 [term 2 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 02:02:22.978639  2200 consensus_queue.cc:1035] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 02:02:22.978993  2693 consensus_queue.cc:1035] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 02:02:22.982312  2200 raft_consensus.cc:2932] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:22.982825  1825 raft_consensus.cc:2932] T b80d547302ec453aaf80fa6b64eebc13 P eb5a2343514740caa85ff0228a535123 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:22.983287  1288 raft_consensus.cc:2932] T b80d547302ec453aaf80fa6b64eebc13 P d37e50431f30463c872c5d819ddf2c25 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:22.988149   867 catalog_manager.cc:5484] T b80d547302ec453aaf80fa6b64eebc13 P eb5a2343514740caa85ff0228a535123 reported cstate change: config changed from index 7 to 8. New cstate: current_term: 2 leader_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:22.998440  1019 consensus_queue.cc:237] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: NON_VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: true } }
I20240502 02:02:23.002213  1288 raft_consensus.cc:1261] T b80d547302ec453aaf80fa6b64eebc13 P d37e50431f30463c872c5d819ddf2c25 [term 2 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
W20240502 02:02:23.002606   953 consensus_peers.cc:485] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f -> Peer 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130:43631): Couldn't send request to peer 8b9e878e72f64482bdd89f28ac427494. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:02:23.003041  1825 raft_consensus.cc:1261] T b80d547302ec453aaf80fa6b64eebc13 P eb5a2343514740caa85ff0228a535123 [term 2 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 02:02:23.003157  2696 consensus_queue.cc:1035] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 02:02:23.003670  2693 consensus_queue.cc:1035] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 02:02:23.004261  1019 consensus_queue.cc:237] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } }
I20240502 02:02:23.005825  2693 raft_consensus.cc:2932] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: NON_VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: true } } }
I20240502 02:02:23.006456  1288 raft_consensus.cc:2932] T b80d547302ec453aaf80fa6b64eebc13 P d37e50431f30463c872c5d819ddf2c25 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: NON_VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: true } } }
I20240502 02:02:23.008739  1825 raft_consensus.cc:1261] T 53d5bc373eb14d44bf1dd07375f40f00 P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 02:02:23.009011   860 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b80d547302ec453aaf80fa6b64eebc13 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 02:02:23.009348  2693 consensus_queue.cc:1035] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 02:02:23.009662  1825 raft_consensus.cc:2932] T b80d547302ec453aaf80fa6b64eebc13 P eb5a2343514740caa85ff0228a535123 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: NON_VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: true } } }
I20240502 02:02:23.011610   866 catalog_manager.cc:5484] T b80d547302ec453aaf80fa6b64eebc13 P d37e50431f30463c872c5d819ddf2c25 reported cstate change: config changed from index 8 to 9, NON_VOTER 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130) added. New cstate: current_term: 2 leader_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: NON_VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: true } } }
I20240502 02:02:23.012472  1289 raft_consensus.cc:1261] T 53d5bc373eb14d44bf1dd07375f40f00 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 02:02:23.014168  2696 consensus_queue.cc:1035] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 02:02:23.016185  2200 raft_consensus.cc:2932] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:23.019657  1825 raft_consensus.cc:2932] T 53d5bc373eb14d44bf1dd07375f40f00 P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:23.021754   866 catalog_manager.cc:5484] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 02:02:23.029215  1288 raft_consensus.cc:2932] T 53d5bc373eb14d44bf1dd07375f40f00 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:23.029704  1019 consensus_queue.cc:237] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: NON_VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: true } }
W20240502 02:02:23.033497   953 consensus_peers.cc:485] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f -> Peer 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130:43631): Couldn't send request to peer 8b9e878e72f64482bdd89f28ac427494. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:02:23.034478  1289 raft_consensus.cc:1261] T 53d5bc373eb14d44bf1dd07375f40f00 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 02:02:23.034497  1825 raft_consensus.cc:1261] T 53d5bc373eb14d44bf1dd07375f40f00 P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 02:02:23.035269  2693 consensus_queue.cc:1035] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 02:02:23.036983  2693 consensus_queue.cc:1035] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 02:02:23.037933  2200 raft_consensus.cc:2932] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: NON_VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: true } } }
I20240502 02:02:23.040014  1289 raft_consensus.cc:2932] T 53d5bc373eb14d44bf1dd07375f40f00 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: NON_VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: true } } }
I20240502 02:02:23.042569   860 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 53d5bc373eb14d44bf1dd07375f40f00 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 02:02:23.044211  1825 raft_consensus.cc:2932] T 53d5bc373eb14d44bf1dd07375f40f00 P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: NON_VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: true } } }
I20240502 02:02:23.044901   866 catalog_manager.cc:5484] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f reported cstate change: config changed from index 7 to 8, NON_VOTER 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130) added. New cstate: current_term: 1 leader_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: NON_VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 02:02:23.063086  1156 consensus_queue.cc:237] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } }
I20240502 02:02:23.065745  1018 raft_consensus.cc:1261] T 2a1ce57d725948239c211b4200d5f037 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Refusing update from remote peer 8b9e878e72f64482bdd89f28ac427494: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 02:02:23.065774  1289 raft_consensus.cc:1261] T 2a1ce57d725948239c211b4200d5f037 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Refusing update from remote peer 8b9e878e72f64482bdd89f28ac427494: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 02:02:23.066361  2717 consensus_queue.cc:1035] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:02:23.066727  2198 consensus_queue.cc:1035] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:02:23.069770  2198 raft_consensus.cc:2932] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } }
I20240502 02:02:23.070515  1018 raft_consensus.cc:2932] T 2a1ce57d725948239c211b4200d5f037 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } }
I20240502 02:02:23.071061  1289 raft_consensus.cc:2932] T 2a1ce57d725948239c211b4200d5f037 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } }
I20240502 02:02:23.074815   868 catalog_manager.cc:5484] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "8b9e878e72f64482bdd89f28ac427494" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } health_report { overall_health: HEALTHY } } }
I20240502 02:02:23.079571  1156 consensus_queue.cc:237] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } }
I20240502 02:02:23.082125  1018 raft_consensus.cc:1261] T 2a1ce57d725948239c211b4200d5f037 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Refusing update from remote peer 8b9e878e72f64482bdd89f28ac427494: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 02:02:23.082557  1089 consensus_peers.cc:485] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494 -> Peer eb5a2343514740caa85ff0228a535123 (127.25.109.132:44441): Couldn't send request to peer eb5a2343514740caa85ff0228a535123. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2a1ce57d725948239c211b4200d5f037. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:02:23.083202  1289 raft_consensus.cc:1261] T 2a1ce57d725948239c211b4200d5f037 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Refusing update from remote peer 8b9e878e72f64482bdd89f28ac427494: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:02:23.083617  2720 consensus_queue.cc:1035] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:23.084065  2198 consensus_queue.cc:1035] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:23.086342  2198 raft_consensus.cc:2932] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER eb5a2343514740caa85ff0228a535123 (127.25.109.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } } }
I20240502 02:02:23.087046  1289 raft_consensus.cc:2932] T 2a1ce57d725948239c211b4200d5f037 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER eb5a2343514740caa85ff0228a535123 (127.25.109.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } } }
I20240502 02:02:23.089998  1018 raft_consensus.cc:2932] T 2a1ce57d725948239c211b4200d5f037 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER eb5a2343514740caa85ff0228a535123 (127.25.109.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } } }
I20240502 02:02:23.092018   861 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2a1ce57d725948239c211b4200d5f037 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 02:02:23.095381   867 catalog_manager.cc:5484] T 2a1ce57d725948239c211b4200d5f037 P d37e50431f30463c872c5d819ddf2c25 reported cstate change: config changed from index 4 to 5, NON_VOTER eb5a2343514740caa85ff0228a535123 (127.25.109.132) added. New cstate: current_term: 1 leader_uuid: "8b9e878e72f64482bdd89f28ac427494" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } } }
I20240502 02:02:23.418578  2751 ts_tablet_manager.cc:902] T 53d5bc373eb14d44bf1dd07375f40f00 P 8b9e878e72f64482bdd89f28ac427494: Initiating tablet copy from peer d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129:33963)
I20240502 02:02:23.420403  2751 tablet_copy_client.cc:308] T 53d5bc373eb14d44bf1dd07375f40f00 P 8b9e878e72f64482bdd89f28ac427494: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.129:33963
I20240502 02:02:23.421347  1040 tablet_copy_service.cc:140] P d0e4135c07ac4f2389ffe4e7379df71f: Received BeginTabletCopySession request for tablet 53d5bc373eb14d44bf1dd07375f40f00 from peer 8b9e878e72f64482bdd89f28ac427494 ({username='slave'} at 127.25.109.130:37059)
I20240502 02:02:23.421516  1040 tablet_copy_service.cc:161] P d0e4135c07ac4f2389ffe4e7379df71f: Beginning new tablet copy session on tablet 53d5bc373eb14d44bf1dd07375f40f00 from peer 8b9e878e72f64482bdd89f28ac427494 at {username='slave'} at 127.25.109.130:37059: session id = 8b9e878e72f64482bdd89f28ac427494-53d5bc373eb14d44bf1dd07375f40f00
I20240502 02:02:23.423513  1040 tablet_copy_source_session.cc:205] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 02:02:23.425021  2751 ts_tablet_manager.cc:1875] T 53d5bc373eb14d44bf1dd07375f40f00 P 8b9e878e72f64482bdd89f28ac427494: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 02:02:23.430207  2751 ts_tablet_manager.cc:1888] T 53d5bc373eb14d44bf1dd07375f40f00 P 8b9e878e72f64482bdd89f28ac427494: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 02:02:23.430505  2751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53d5bc373eb14d44bf1dd07375f40f00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:23.433017  2751 tablet_copy_client.cc:783] T 53d5bc373eb14d44bf1dd07375f40f00 P 8b9e878e72f64482bdd89f28ac427494: tablet copy: Starting download of 0 data blocks...
I20240502 02:02:23.433323  2751 tablet_copy_client.cc:647] T 53d5bc373eb14d44bf1dd07375f40f00 P 8b9e878e72f64482bdd89f28ac427494: tablet copy: Starting download of 1 WAL segments...
I20240502 02:02:23.435418  2751 tablet_copy_client.cc:523] T 53d5bc373eb14d44bf1dd07375f40f00 P 8b9e878e72f64482bdd89f28ac427494: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:02:23.438447  2751 tablet_bootstrap.cc:492] T 53d5bc373eb14d44bf1dd07375f40f00 P 8b9e878e72f64482bdd89f28ac427494: Bootstrap starting.
I20240502 02:02:23.472730  2751 tablet_bootstrap.cc:492] T 53d5bc373eb14d44bf1dd07375f40f00 P 8b9e878e72f64482bdd89f28ac427494: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:02:23.473166  2751 tablet_bootstrap.cc:492] T 53d5bc373eb14d44bf1dd07375f40f00 P 8b9e878e72f64482bdd89f28ac427494: Bootstrap complete.
I20240502 02:02:23.473433  2751 ts_tablet_manager.cc:1367] T 53d5bc373eb14d44bf1dd07375f40f00 P 8b9e878e72f64482bdd89f28ac427494: Time spent bootstrapping tablet: real 0.035s	user 0.032s	sys 0.004s
I20240502 02:02:23.474112  2751 raft_consensus.cc:348] T 53d5bc373eb14d44bf1dd07375f40f00 P 8b9e878e72f64482bdd89f28ac427494 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: NON_VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: true } }
I20240502 02:02:23.474349  2751 raft_consensus.cc:729] T 53d5bc373eb14d44bf1dd07375f40f00 P 8b9e878e72f64482bdd89f28ac427494 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8b9e878e72f64482bdd89f28ac427494, State: Initialized, Role: LEARNER
I20240502 02:02:23.474536  2751 consensus_queue.cc:260] T 53d5bc373eb14d44bf1dd07375f40f00 P 8b9e878e72f64482bdd89f28ac427494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: NON_VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: true } }
I20240502 02:02:23.475093  2751 ts_tablet_manager.cc:1398] T 53d5bc373eb14d44bf1dd07375f40f00 P 8b9e878e72f64482bdd89f28ac427494: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 02:02:23.476153  1040 tablet_copy_service.cc:342] P d0e4135c07ac4f2389ffe4e7379df71f: Request end of tablet copy session 8b9e878e72f64482bdd89f28ac427494-53d5bc373eb14d44bf1dd07375f40f00 received from {username='slave'} at 127.25.109.130:37059
I20240502 02:02:23.476346  1040 tablet_copy_service.cc:433] P d0e4135c07ac4f2389ffe4e7379df71f: ending tablet copy session 8b9e878e72f64482bdd89f28ac427494-53d5bc373eb14d44bf1dd07375f40f00 on tablet 53d5bc373eb14d44bf1dd07375f40f00 with peer 8b9e878e72f64482bdd89f28ac427494
I20240502 02:02:23.545753  2751 ts_tablet_manager.cc:902] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494: Initiating tablet copy from peer d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129:33963)
I20240502 02:02:23.546905  2751 tablet_copy_client.cc:308] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.129:33963
I20240502 02:02:23.547653  1040 tablet_copy_service.cc:140] P d0e4135c07ac4f2389ffe4e7379df71f: Received BeginTabletCopySession request for tablet b80d547302ec453aaf80fa6b64eebc13 from peer 8b9e878e72f64482bdd89f28ac427494 ({username='slave'} at 127.25.109.130:37059)
I20240502 02:02:23.547837  1040 tablet_copy_service.cc:161] P d0e4135c07ac4f2389ffe4e7379df71f: Beginning new tablet copy session on tablet b80d547302ec453aaf80fa6b64eebc13 from peer 8b9e878e72f64482bdd89f28ac427494 at {username='slave'} at 127.25.109.130:37059: session id = 8b9e878e72f64482bdd89f28ac427494-b80d547302ec453aaf80fa6b64eebc13
I20240502 02:02:23.550011  1040 tablet_copy_source_session.cc:205] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 02:02:23.551156  2751 ts_tablet_manager.cc:1875] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 02:02:23.556149  2751 ts_tablet_manager.cc:1888] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.6
I20240502 02:02:23.556476  2751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b80d547302ec453aaf80fa6b64eebc13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:23.558990  2751 tablet_copy_client.cc:783] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494: tablet copy: Starting download of 0 data blocks...
I20240502 02:02:23.559250  2751 tablet_copy_client.cc:647] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494: tablet copy: Starting download of 1 WAL segments...
I20240502 02:02:23.561414  2751 tablet_copy_client.cc:523] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:02:23.564517  2751 tablet_bootstrap.cc:492] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494: Bootstrap starting.
I20240502 02:02:23.600198  2751 tablet_bootstrap.cc:492] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:02:23.600680  2751 tablet_bootstrap.cc:492] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494: Bootstrap complete.
I20240502 02:02:23.600931  2751 ts_tablet_manager.cc:1367] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494: Time spent bootstrapping tablet: real 0.037s	user 0.031s	sys 0.005s
I20240502 02:02:23.601663  2751 raft_consensus.cc:348] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: NON_VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: true } }
I20240502 02:02:23.601866  2751 raft_consensus.cc:729] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 8b9e878e72f64482bdd89f28ac427494, State: Initialized, Role: LEARNER
I20240502 02:02:23.602064  2751 consensus_queue.cc:260] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: NON_VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: true } }
I20240502 02:02:23.602509  2751 ts_tablet_manager.cc:1398] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 02:02:23.603338  1040 tablet_copy_service.cc:342] P d0e4135c07ac4f2389ffe4e7379df71f: Request end of tablet copy session 8b9e878e72f64482bdd89f28ac427494-b80d547302ec453aaf80fa6b64eebc13 received from {username='slave'} at 127.25.109.130:37059
I20240502 02:02:23.603461  1040 tablet_copy_service.cc:433] P d0e4135c07ac4f2389ffe4e7379df71f: ending tablet copy session 8b9e878e72f64482bdd89f28ac427494-b80d547302ec453aaf80fa6b64eebc13 on tablet b80d547302ec453aaf80fa6b64eebc13 with peer 8b9e878e72f64482bdd89f28ac427494
I20240502 02:02:23.616779  2779 ts_tablet_manager.cc:902] T 2a1ce57d725948239c211b4200d5f037 P eb5a2343514740caa85ff0228a535123: Initiating tablet copy from peer 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130:43631)
I20240502 02:02:23.618023  2779 tablet_copy_client.cc:308] T 2a1ce57d725948239c211b4200d5f037 P eb5a2343514740caa85ff0228a535123: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.130:43631
I20240502 02:02:23.618870  1176 tablet_copy_service.cc:140] P 8b9e878e72f64482bdd89f28ac427494: Received BeginTabletCopySession request for tablet 2a1ce57d725948239c211b4200d5f037 from peer eb5a2343514740caa85ff0228a535123 ({username='slave'} at 127.25.109.132:52325)
I20240502 02:02:23.619036  1176 tablet_copy_service.cc:161] P 8b9e878e72f64482bdd89f28ac427494: Beginning new tablet copy session on tablet 2a1ce57d725948239c211b4200d5f037 from peer eb5a2343514740caa85ff0228a535123 at {username='slave'} at 127.25.109.132:52325: session id = eb5a2343514740caa85ff0228a535123-2a1ce57d725948239c211b4200d5f037
I20240502 02:02:23.621282  1176 tablet_copy_source_session.cc:205] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 02:02:23.622632  2779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a1ce57d725948239c211b4200d5f037. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:23.632370  2779 tablet_copy_client.cc:783] T 2a1ce57d725948239c211b4200d5f037 P eb5a2343514740caa85ff0228a535123: tablet copy: Starting download of 0 data blocks...
I20240502 02:02:23.632742  2779 tablet_copy_client.cc:647] T 2a1ce57d725948239c211b4200d5f037 P eb5a2343514740caa85ff0228a535123: tablet copy: Starting download of 1 WAL segments...
I20240502 02:02:23.635700  2779 tablet_copy_client.cc:523] T 2a1ce57d725948239c211b4200d5f037 P eb5a2343514740caa85ff0228a535123: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:02:23.640408  2779 tablet_bootstrap.cc:492] T 2a1ce57d725948239c211b4200d5f037 P eb5a2343514740caa85ff0228a535123: Bootstrap starting.
I20240502 02:02:23.677578  2779 tablet_bootstrap.cc:492] T 2a1ce57d725948239c211b4200d5f037 P eb5a2343514740caa85ff0228a535123: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:02:23.678012  2779 tablet_bootstrap.cc:492] T 2a1ce57d725948239c211b4200d5f037 P eb5a2343514740caa85ff0228a535123: Bootstrap complete.
I20240502 02:02:23.678297  2779 ts_tablet_manager.cc:1367] T 2a1ce57d725948239c211b4200d5f037 P eb5a2343514740caa85ff0228a535123: Time spent bootstrapping tablet: real 0.038s	user 0.035s	sys 0.000s
I20240502 02:02:23.679039  2779 raft_consensus.cc:348] T 2a1ce57d725948239c211b4200d5f037 P eb5a2343514740caa85ff0228a535123 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } }
I20240502 02:02:23.679277  2779 raft_consensus.cc:729] T 2a1ce57d725948239c211b4200d5f037 P eb5a2343514740caa85ff0228a535123 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb5a2343514740caa85ff0228a535123, State: Initialized, Role: LEARNER
I20240502 02:02:23.679476  2779 consensus_queue.cc:260] T 2a1ce57d725948239c211b4200d5f037 P eb5a2343514740caa85ff0228a535123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } }
I20240502 02:02:23.680505  2779 ts_tablet_manager.cc:1398] T 2a1ce57d725948239c211b4200d5f037 P eb5a2343514740caa85ff0228a535123: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.001s
I20240502 02:02:23.681373  1176 tablet_copy_service.cc:342] P 8b9e878e72f64482bdd89f28ac427494: Request end of tablet copy session eb5a2343514740caa85ff0228a535123-2a1ce57d725948239c211b4200d5f037 received from {username='slave'} at 127.25.109.132:52325
I20240502 02:02:23.681542  1176 tablet_copy_service.cc:433] P 8b9e878e72f64482bdd89f28ac427494: ending tablet copy session eb5a2343514740caa85ff0228a535123-2a1ce57d725948239c211b4200d5f037 on tablet 2a1ce57d725948239c211b4200d5f037 with peer eb5a2343514740caa85ff0228a535123
I20240502 02:02:23.971787  1156 raft_consensus.cc:1203] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9]   Dedup: 2.9->[]
I20240502 02:02:23.974818  1156 raft_consensus.cc:1203] T 53d5bc373eb14d44bf1dd07375f40f00 P 8b9e878e72f64482bdd89f28ac427494 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240502 02:02:24.187361  1825 raft_consensus.cc:1203] T 2a1ce57d725948239c211b4200d5f037 P eb5a2343514740caa85ff0228a535123 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 02:02:24.353708  2803 raft_consensus.cc:1050] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f: attempting to promote NON_VOTER 8b9e878e72f64482bdd89f28ac427494 to VOTER
I20240502 02:02:24.354512  2803 consensus_queue.cc:237] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: false } }
I20240502 02:02:24.357093  1825 raft_consensus.cc:1261] T b80d547302ec453aaf80fa6b64eebc13 P eb5a2343514740caa85ff0228a535123 [term 2 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 02:02:24.357143  1289 raft_consensus.cc:1261] T b80d547302ec453aaf80fa6b64eebc13 P d37e50431f30463c872c5d819ddf2c25 [term 2 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 02:02:24.357183  1156 raft_consensus.cc:1261] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [term 2 LEARNER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 02:02:24.357829  2696 consensus_queue.cc:1035] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 02:02:24.358194  2803 consensus_queue.cc:1035] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 02:02:24.358541  2838 consensus_queue.cc:1035] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 02:02:24.375340  2803 raft_consensus.cc:2932] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: false } } }
I20240502 02:02:24.376338  1156 raft_consensus.cc:2932] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: false } } }
I20240502 02:02:24.376536  1825 raft_consensus.cc:2932] T b80d547302ec453aaf80fa6b64eebc13 P eb5a2343514740caa85ff0228a535123 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: false } } }
I20240502 02:02:24.380358   873 catalog_manager.cc:5484] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f reported cstate change: config changed from index 9 to 10, 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 02:02:24.382547  1289 raft_consensus.cc:2932] T b80d547302ec453aaf80fa6b64eebc13 P d37e50431f30463c872c5d819ddf2c25 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: false } } }
I20240502 02:02:24.388283  1018 consensus_queue.cc:237] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: false } }
I20240502 02:02:24.388689  2696 raft_consensus.cc:1050] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f: attempting to promote NON_VOTER 8b9e878e72f64482bdd89f28ac427494 to VOTER
I20240502 02:02:24.389509  2696 consensus_queue.cc:237] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: false } }
I20240502 02:02:24.391273  1156 raft_consensus.cc:1261] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [term 2 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 02:02:24.391685  1825 raft_consensus.cc:1261] T b80d547302ec453aaf80fa6b64eebc13 P eb5a2343514740caa85ff0228a535123 [term 2 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 02:02:24.392248  2838 consensus_queue.cc:1035] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 02:02:24.392555  2803 consensus_queue.cc:1035] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 02:02:24.395259  2803 raft_consensus.cc:2932] T b80d547302ec453aaf80fa6b64eebc13 P d0e4135c07ac4f2389ffe4e7379df71f [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER d37e50431f30463c872c5d819ddf2c25 (127.25.109.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: false } } }
I20240502 02:02:24.396828  1156 raft_consensus.cc:2932] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER d37e50431f30463c872c5d819ddf2c25 (127.25.109.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: false } } }
I20240502 02:02:24.400000  1824 raft_consensus.cc:1261] T 53d5bc373eb14d44bf1dd07375f40f00 P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 02:02:24.400705  1155 raft_consensus.cc:1261] T 53d5bc373eb14d44bf1dd07375f40f00 P 8b9e878e72f64482bdd89f28ac427494 [term 1 LEARNER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 02:02:24.401558  2802 consensus_queue.cc:1035] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 02:02:24.401398  1825 raft_consensus.cc:2932] T b80d547302ec453aaf80fa6b64eebc13 P eb5a2343514740caa85ff0228a535123 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER d37e50431f30463c872c5d819ddf2c25 (127.25.109.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: false } } }
I20240502 02:02:24.401779  1289 raft_consensus.cc:1261] T 53d5bc373eb14d44bf1dd07375f40f00 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 02:02:24.402392  2802 consensus_queue.cc:1035] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 02:02:24.402889  2838 consensus_queue.cc:1035] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 02:02:24.402930   867 catalog_manager.cc:5484] T b80d547302ec453aaf80fa6b64eebc13 P 8b9e878e72f64482bdd89f28ac427494 reported cstate change: config changed from index 10 to 11, VOTER d37e50431f30463c872c5d819ddf2c25 (127.25.109.131) evicted. New cstate: current_term: 2 leader_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: false } } }
I20240502 02:02:24.406924   860 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b80d547302ec453aaf80fa6b64eebc13 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 02:02:24.409703  1269 tablet_service.cc:1514] Processing DeleteTablet for tablet b80d547302ec453aaf80fa6b64eebc13 with delete_type TABLET_DATA_TOMBSTONED (TS d37e50431f30463c872c5d819ddf2c25 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:55198
I20240502 02:02:24.411196  2851 tablet_replica.cc:330] T b80d547302ec453aaf80fa6b64eebc13 P d37e50431f30463c872c5d819ddf2c25: stopping tablet replica
I20240502 02:02:24.411726  2851 raft_consensus.cc:2229] T b80d547302ec453aaf80fa6b64eebc13 P d37e50431f30463c872c5d819ddf2c25 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 02:02:24.412209  2851 raft_consensus.cc:2258] T b80d547302ec453aaf80fa6b64eebc13 P d37e50431f30463c872c5d819ddf2c25 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 02:02:24.414124  2851 ts_tablet_manager.cc:1875] T b80d547302ec453aaf80fa6b64eebc13 P d37e50431f30463c872c5d819ddf2c25: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 02:02:24.418118  2802 raft_consensus.cc:2932] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: false } } }
I20240502 02:02:24.421480  2851 ts_tablet_manager.cc:1888] T b80d547302ec453aaf80fa6b64eebc13 P d37e50431f30463c872c5d819ddf2c25: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240502 02:02:24.421643  2851 log.cc:1196] T b80d547302ec453aaf80fa6b64eebc13 P d37e50431f30463c872c5d819ddf2c25: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/wals/b80d547302ec453aaf80fa6b64eebc13
I20240502 02:02:24.422673   859 catalog_manager.cc:4830] TS d37e50431f30463c872c5d819ddf2c25 (127.25.109.131:37799): tablet b80d547302ec453aaf80fa6b64eebc13 (table rebalance_test_table_0 [id=3920dc53a6a846338bdd42178520e7af]) successfully deleted
I20240502 02:02:24.423481  1289 raft_consensus.cc:2932] T 53d5bc373eb14d44bf1dd07375f40f00 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: false } } }
I20240502 02:02:24.426015   873 catalog_manager.cc:5484] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f reported cstate change: config changed from index 8 to 9, 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 02:02:24.426767  1155 raft_consensus.cc:2932] T 53d5bc373eb14d44bf1dd07375f40f00 P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: false } } }
I20240502 02:02:24.427171  1824 raft_consensus.cc:2932] T 53d5bc373eb14d44bf1dd07375f40f00 P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: false } } }
I20240502 02:02:24.434227  1018 consensus_queue.cc:237] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: false } }
I20240502 02:02:24.436038  1824 raft_consensus.cc:1261] T 53d5bc373eb14d44bf1dd07375f40f00 P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 02:02:24.436172  1155 raft_consensus.cc:1261] T 53d5bc373eb14d44bf1dd07375f40f00 P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 02:02:24.436666  2803 consensus_queue.cc:1035] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 02:02:24.437031  2838 consensus_queue.cc:1035] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 02:02:24.439687  2847 raft_consensus.cc:2932] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d37e50431f30463c872c5d819ddf2c25 (127.25.109.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: false } } }
I20240502 02:02:24.440280  1155 raft_consensus.cc:2932] T 53d5bc373eb14d44bf1dd07375f40f00 P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d37e50431f30463c872c5d819ddf2c25 (127.25.109.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: false } } }
I20240502 02:02:24.440279  1824 raft_consensus.cc:2932] T 53d5bc373eb14d44bf1dd07375f40f00 P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d37e50431f30463c872c5d819ddf2c25 (127.25.109.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: false } } }
I20240502 02:02:24.442901   860 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 53d5bc373eb14d44bf1dd07375f40f00 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 02:02:24.445030   868 catalog_manager.cc:5484] T 53d5bc373eb14d44bf1dd07375f40f00 P d0e4135c07ac4f2389ffe4e7379df71f reported cstate change: config changed from index 9 to 10, VOTER d37e50431f30463c872c5d819ddf2c25 (127.25.109.131) evicted. New cstate: current_term: 1 leader_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 02:02:24.448913  1269 tablet_service.cc:1514] Processing DeleteTablet for tablet 53d5bc373eb14d44bf1dd07375f40f00 with delete_type TABLET_DATA_TOMBSTONED (TS d37e50431f30463c872c5d819ddf2c25 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:55198
I20240502 02:02:24.449270  2851 tablet_replica.cc:330] T 53d5bc373eb14d44bf1dd07375f40f00 P d37e50431f30463c872c5d819ddf2c25: stopping tablet replica
I20240502 02:02:24.449448  2851 raft_consensus.cc:2229] T 53d5bc373eb14d44bf1dd07375f40f00 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 02:02:24.449734  2851 raft_consensus.cc:2258] T 53d5bc373eb14d44bf1dd07375f40f00 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 02:02:24.450978  2851 ts_tablet_manager.cc:1875] T 53d5bc373eb14d44bf1dd07375f40f00 P d37e50431f30463c872c5d819ddf2c25: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 02:02:24.455595  2851 ts_tablet_manager.cc:1888] T 53d5bc373eb14d44bf1dd07375f40f00 P d37e50431f30463c872c5d819ddf2c25: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 02:02:24.455729  2851 log.cc:1196] T 53d5bc373eb14d44bf1dd07375f40f00 P d37e50431f30463c872c5d819ddf2c25: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/wals/53d5bc373eb14d44bf1dd07375f40f00
I20240502 02:02:24.456538   859 catalog_manager.cc:4830] TS d37e50431f30463c872c5d819ddf2c25 (127.25.109.131:37799): tablet 53d5bc373eb14d44bf1dd07375f40f00 (table rebalance_test_table_2 [id=57c9d0a8579e4e908683ee50f24d5aa1]) successfully deleted
I20240502 02:02:24.637058  2819 raft_consensus.cc:1050] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494: attempting to promote NON_VOTER eb5a2343514740caa85ff0228a535123 to VOTER
I20240502 02:02:24.637840  2819 consensus_queue.cc:237] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } }
I20240502 02:02:24.640457  1018 raft_consensus.cc:1261] T 2a1ce57d725948239c211b4200d5f037 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Refusing update from remote peer 8b9e878e72f64482bdd89f28ac427494: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:02:24.640477  1289 raft_consensus.cc:1261] T 2a1ce57d725948239c211b4200d5f037 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Refusing update from remote peer 8b9e878e72f64482bdd89f28ac427494: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:02:24.640877  1824 raft_consensus.cc:1261] T 2a1ce57d725948239c211b4200d5f037 P eb5a2343514740caa85ff0228a535123 [term 1 LEARNER]: Refusing update from remote peer 8b9e878e72f64482bdd89f28ac427494: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:02:24.641124  2820 consensus_queue.cc:1035] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:02:24.641530  2868 consensus_queue.cc:1035] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:02:24.641852  2819 consensus_queue.cc:1035] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:02:24.653645  2819 raft_consensus.cc:2932] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, eb5a2343514740caa85ff0228a535123 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:24.654495  1289 raft_consensus.cc:2932] T 2a1ce57d725948239c211b4200d5f037 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, eb5a2343514740caa85ff0228a535123 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:24.654479  1018 raft_consensus.cc:2932] T 2a1ce57d725948239c211b4200d5f037 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, eb5a2343514740caa85ff0228a535123 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:24.657653  1824 raft_consensus.cc:2932] T 2a1ce57d725948239c211b4200d5f037 P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, eb5a2343514740caa85ff0228a535123 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:24.658702   873 catalog_manager.cc:5484] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494 reported cstate change: config changed from index 5 to 6, eb5a2343514740caa85ff0228a535123 (127.25.109.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8b9e878e72f64482bdd89f28ac427494" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 02:02:24.664214  1155 consensus_queue.cc:237] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } }
I20240502 02:02:24.666039  1018 raft_consensus.cc:1261] T 2a1ce57d725948239c211b4200d5f037 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Refusing update from remote peer 8b9e878e72f64482bdd89f28ac427494: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 02:02:24.666199  1824 raft_consensus.cc:1261] T 2a1ce57d725948239c211b4200d5f037 P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Refusing update from remote peer 8b9e878e72f64482bdd89f28ac427494: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 02:02:24.666776  2819 consensus_queue.cc:1035] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 02:02:24.667052  2868 consensus_queue.cc:1035] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 02:02:24.669198  2820 raft_consensus.cc:2932] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d37e50431f30463c872c5d819ddf2c25 (127.25.109.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:24.669845  1018 raft_consensus.cc:2932] T 2a1ce57d725948239c211b4200d5f037 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d37e50431f30463c872c5d819ddf2c25 (127.25.109.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:24.672190   861 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2a1ce57d725948239c211b4200d5f037 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 02:02:24.673733  1824 raft_consensus.cc:2932] T 2a1ce57d725948239c211b4200d5f037 P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d37e50431f30463c872c5d819ddf2c25 (127.25.109.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:24.674015   867 catalog_manager.cc:5484] T 2a1ce57d725948239c211b4200d5f037 P 8b9e878e72f64482bdd89f28ac427494 reported cstate change: config changed from index 6 to 7, VOTER d37e50431f30463c872c5d819ddf2c25 (127.25.109.131) evicted. New cstate: current_term: 1 leader_uuid: "8b9e878e72f64482bdd89f28ac427494" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 02:02:24.678179  1269 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a1ce57d725948239c211b4200d5f037 with delete_type TABLET_DATA_TOMBSTONED (TS d37e50431f30463c872c5d819ddf2c25 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:55198
I20240502 02:02:24.678589  2851 tablet_replica.cc:330] T 2a1ce57d725948239c211b4200d5f037 P d37e50431f30463c872c5d819ddf2c25: stopping tablet replica
I20240502 02:02:24.678791  2851 raft_consensus.cc:2229] T 2a1ce57d725948239c211b4200d5f037 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 02:02:24.679042  2851 raft_consensus.cc:2258] T 2a1ce57d725948239c211b4200d5f037 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 02:02:24.680099  2851 ts_tablet_manager.cc:1875] T 2a1ce57d725948239c211b4200d5f037 P d37e50431f30463c872c5d819ddf2c25: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 02:02:24.685514  2851 ts_tablet_manager.cc:1888] T 2a1ce57d725948239c211b4200d5f037 P d37e50431f30463c872c5d819ddf2c25: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 02:02:24.685633  2851 log.cc:1196] T 2a1ce57d725948239c211b4200d5f037 P d37e50431f30463c872c5d819ddf2c25: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/wals/2a1ce57d725948239c211b4200d5f037
I20240502 02:02:24.686303   859 catalog_manager.cc:4830] TS d37e50431f30463c872c5d819ddf2c25 (127.25.109.131:37799): tablet 2a1ce57d725948239c211b4200d5f037 (table rebalance_test_table_0 [id=3920dc53a6a846338bdd42178520e7af]) successfully deleted
I20240502 02:02:24.704813  1155 consensus_queue.cc:237] T 8573fb0d09af421ebbaeb98759e85f4d P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } }
I20240502 02:02:24.706701  1289 raft_consensus.cc:1261] T 8573fb0d09af421ebbaeb98759e85f4d P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Refusing update from remote peer 8b9e878e72f64482bdd89f28ac427494: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 02:02:24.706753  1018 raft_consensus.cc:1261] T 8573fb0d09af421ebbaeb98759e85f4d P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Refusing update from remote peer 8b9e878e72f64482bdd89f28ac427494: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 02:02:24.707279  2820 consensus_queue.cc:1035] T 8573fb0d09af421ebbaeb98759e85f4d P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:02:24.707544  2819 consensus_queue.cc:1035] T 8573fb0d09af421ebbaeb98759e85f4d P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:02:24.710090  2717 raft_consensus.cc:2932] T 8573fb0d09af421ebbaeb98759e85f4d P 8b9e878e72f64482bdd89f28ac427494 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } }
I20240502 02:02:24.710768  1018 raft_consensus.cc:2932] T 8573fb0d09af421ebbaeb98759e85f4d P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } }
I20240502 02:02:24.713284  1289 raft_consensus.cc:2932] T 8573fb0d09af421ebbaeb98759e85f4d P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } }
I20240502 02:02:24.714718   868 catalog_manager.cc:5484] T 8573fb0d09af421ebbaeb98759e85f4d P 8b9e878e72f64482bdd89f28ac427494 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "8b9e878e72f64482bdd89f28ac427494" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } health_report { overall_health: HEALTHY } } }
I20240502 02:02:24.720724  1155 consensus_queue.cc:237] T 8573fb0d09af421ebbaeb98759e85f4d P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } }
W20240502 02:02:24.722865  1089 consensus_peers.cc:485] T 8573fb0d09af421ebbaeb98759e85f4d P 8b9e878e72f64482bdd89f28ac427494 -> Peer eb5a2343514740caa85ff0228a535123 (127.25.109.132:44441): Couldn't send request to peer eb5a2343514740caa85ff0228a535123. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8573fb0d09af421ebbaeb98759e85f4d. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:02:24.723727  1018 raft_consensus.cc:1261] T 8573fb0d09af421ebbaeb98759e85f4d P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Refusing update from remote peer 8b9e878e72f64482bdd89f28ac427494: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:02:24.724022  1289 raft_consensus.cc:1261] T 8573fb0d09af421ebbaeb98759e85f4d P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Refusing update from remote peer 8b9e878e72f64482bdd89f28ac427494: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:02:24.724577  2717 consensus_queue.cc:1035] T 8573fb0d09af421ebbaeb98759e85f4d P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:24.724906  2820 consensus_queue.cc:1035] T 8573fb0d09af421ebbaeb98759e85f4d P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:24.727226  2717 raft_consensus.cc:2932] T 8573fb0d09af421ebbaeb98759e85f4d P 8b9e878e72f64482bdd89f28ac427494 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER eb5a2343514740caa85ff0228a535123 (127.25.109.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } } }
I20240502 02:02:24.727941  1018 raft_consensus.cc:2932] T 8573fb0d09af421ebbaeb98759e85f4d P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER eb5a2343514740caa85ff0228a535123 (127.25.109.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } } }
I20240502 02:02:24.731539   861 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8573fb0d09af421ebbaeb98759e85f4d with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 02:02:24.733161   868 catalog_manager.cc:5484] T 8573fb0d09af421ebbaeb98759e85f4d P 8b9e878e72f64482bdd89f28ac427494 reported cstate change: config changed from index 4 to 5, NON_VOTER eb5a2343514740caa85ff0228a535123 (127.25.109.132) added. New cstate: current_term: 1 leader_uuid: "8b9e878e72f64482bdd89f28ac427494" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 02:02:24.733870  1289 raft_consensus.cc:2932] T 8573fb0d09af421ebbaeb98759e85f4d P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER eb5a2343514740caa85ff0228a535123 (127.25.109.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } } }
I20240502 02:02:24.743781  1018 consensus_queue.cc:237] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } }
I20240502 02:02:24.745832  1289 raft_consensus.cc:1261] T f55670daadbc4a7f88971471629c793f P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 02:02:24.746119  1824 raft_consensus.cc:1261] T f55670daadbc4a7f88971471629c793f P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 02:02:24.746364  2847 consensus_queue.cc:1035] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 02:02:24.746712  2838 consensus_queue.cc:1035] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 02:02:24.749759  2803 raft_consensus.cc:2932] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:24.750501  1289 raft_consensus.cc:2932] T f55670daadbc4a7f88971471629c793f P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:24.750573  1824 raft_consensus.cc:2932] T f55670daadbc4a7f88971471629c793f P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:24.754419   867 catalog_manager.cc:5484] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 02:02:24.759629  1018 consensus_queue.cc:237] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: NON_VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: true } }
W20240502 02:02:24.762600   953 consensus_peers.cc:485] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f -> Peer 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130:43631): Couldn't send request to peer 8b9e878e72f64482bdd89f28ac427494. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:02:24.762974  1289 raft_consensus.cc:1261] T f55670daadbc4a7f88971471629c793f P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 02:02:24.763588  2838 consensus_queue.cc:1035] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 02:02:24.764343  1824 raft_consensus.cc:1261] T f55670daadbc4a7f88971471629c793f P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 02:02:24.765627  2838 consensus_queue.cc:1035] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 02:02:24.766780  2802 raft_consensus.cc:2932] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: NON_VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: true } } }
I20240502 02:02:24.767395  1289 raft_consensus.cc:2932] T f55670daadbc4a7f88971471629c793f P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: NON_VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: true } } }
I20240502 02:02:24.769102  1824 raft_consensus.cc:2932] T f55670daadbc4a7f88971471629c793f P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: NON_VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: true } } }
I20240502 02:02:24.775415   868 catalog_manager.cc:5484] T f55670daadbc4a7f88971471629c793f P d37e50431f30463c872c5d819ddf2c25 reported cstate change: config changed from index 7 to 8, NON_VOTER 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130) added. New cstate: current_term: 1 leader_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: NON_VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: true } } }
I20240502 02:02:24.778170   860 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f55670daadbc4a7f88971471629c793f with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 02:02:25.009769  1018 consensus_queue.cc:237] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } }
I20240502 02:02:25.011948  1155 raft_consensus.cc:1261] T 11210087d9164d2c9c1303e659230192 P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 02:02:25.011957  1289 raft_consensus.cc:1261] T 11210087d9164d2c9c1303e659230192 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 02:02:25.012545  2838 consensus_queue.cc:1035] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:02:25.012871  2802 consensus_queue.cc:1035] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 02:02:25.015723  2847 raft_consensus.cc:2932] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } }
I20240502 02:02:25.016551  1155 raft_consensus.cc:2932] T 11210087d9164d2c9c1303e659230192 P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } }
I20240502 02:02:25.016506  1289 raft_consensus.cc:2932] T 11210087d9164d2c9c1303e659230192 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } }
I20240502 02:02:25.020815   868 catalog_manager.cc:5484] T 11210087d9164d2c9c1303e659230192 P 8b9e878e72f64482bdd89f28ac427494 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } }
I20240502 02:02:25.027531  1018 consensus_queue.cc:237] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } }
I20240502 02:02:25.030177  1289 raft_consensus.cc:1261] T 11210087d9164d2c9c1303e659230192 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 02:02:25.030725   954 consensus_peers.cc:485] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f -> Peer eb5a2343514740caa85ff0228a535123 (127.25.109.132:44441): Couldn't send request to peer eb5a2343514740caa85ff0228a535123. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 11210087d9164d2c9c1303e659230192. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:02:25.031232  2803 consensus_queue.cc:1035] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:25.032095  1155 raft_consensus.cc:1261] T 11210087d9164d2c9c1303e659230192 P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 02:02:25.032941  2838 consensus_queue.cc:1035] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 02:02:25.033768  2696 raft_consensus.cc:2932] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER eb5a2343514740caa85ff0228a535123 (127.25.109.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } } }
I20240502 02:02:25.034368  1289 raft_consensus.cc:2932] T 11210087d9164d2c9c1303e659230192 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER eb5a2343514740caa85ff0228a535123 (127.25.109.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } } }
I20240502 02:02:25.037961   860 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 11210087d9164d2c9c1303e659230192 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 02:02:25.038175  1155 raft_consensus.cc:2932] T 11210087d9164d2c9c1303e659230192 P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER eb5a2343514740caa85ff0228a535123 (127.25.109.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } } }
I20240502 02:02:25.039208   873 catalog_manager.cc:5484] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f reported cstate change: config changed from index 4 to 5, NON_VOTER eb5a2343514740caa85ff0228a535123 (127.25.109.132) added. New cstate: current_term: 1 leader_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 02:02:25.264400  2927 ts_tablet_manager.cc:902] T 8573fb0d09af421ebbaeb98759e85f4d P eb5a2343514740caa85ff0228a535123: Initiating tablet copy from peer 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130:43631)
I20240502 02:02:25.269752  2927 tablet_copy_client.cc:308] T 8573fb0d09af421ebbaeb98759e85f4d P eb5a2343514740caa85ff0228a535123: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.130:43631
I20240502 02:02:25.270646  1176 tablet_copy_service.cc:140] P 8b9e878e72f64482bdd89f28ac427494: Received BeginTabletCopySession request for tablet 8573fb0d09af421ebbaeb98759e85f4d from peer eb5a2343514740caa85ff0228a535123 ({username='slave'} at 127.25.109.132:52325)
I20240502 02:02:25.270864  1176 tablet_copy_service.cc:161] P 8b9e878e72f64482bdd89f28ac427494: Beginning new tablet copy session on tablet 8573fb0d09af421ebbaeb98759e85f4d from peer eb5a2343514740caa85ff0228a535123 at {username='slave'} at 127.25.109.132:52325: session id = eb5a2343514740caa85ff0228a535123-8573fb0d09af421ebbaeb98759e85f4d
I20240502 02:02:25.273541  1176 tablet_copy_source_session.cc:205] T 8573fb0d09af421ebbaeb98759e85f4d P 8b9e878e72f64482bdd89f28ac427494: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 02:02:25.274904  2927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8573fb0d09af421ebbaeb98759e85f4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:25.280963  2927 tablet_copy_client.cc:783] T 8573fb0d09af421ebbaeb98759e85f4d P eb5a2343514740caa85ff0228a535123: tablet copy: Starting download of 0 data blocks...
I20240502 02:02:25.281232  2927 tablet_copy_client.cc:647] T 8573fb0d09af421ebbaeb98759e85f4d P eb5a2343514740caa85ff0228a535123: tablet copy: Starting download of 1 WAL segments...
I20240502 02:02:25.283354  2927 tablet_copy_client.cc:523] T 8573fb0d09af421ebbaeb98759e85f4d P eb5a2343514740caa85ff0228a535123: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:02:25.285832  2927 tablet_bootstrap.cc:492] T 8573fb0d09af421ebbaeb98759e85f4d P eb5a2343514740caa85ff0228a535123: Bootstrap starting.
I20240502 02:02:25.296672  2945 ts_tablet_manager.cc:902] T f55670daadbc4a7f88971471629c793f P 8b9e878e72f64482bdd89f28ac427494: Initiating tablet copy from peer d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129:33963)
I20240502 02:02:25.309177  2945 tablet_copy_client.cc:308] T f55670daadbc4a7f88971471629c793f P 8b9e878e72f64482bdd89f28ac427494: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.129:33963
I20240502 02:02:25.310286  1040 tablet_copy_service.cc:140] P d0e4135c07ac4f2389ffe4e7379df71f: Received BeginTabletCopySession request for tablet f55670daadbc4a7f88971471629c793f from peer 8b9e878e72f64482bdd89f28ac427494 ({username='slave'} at 127.25.109.130:37059)
I20240502 02:02:25.310492  1040 tablet_copy_service.cc:161] P d0e4135c07ac4f2389ffe4e7379df71f: Beginning new tablet copy session on tablet f55670daadbc4a7f88971471629c793f from peer 8b9e878e72f64482bdd89f28ac427494 at {username='slave'} at 127.25.109.130:37059: session id = 8b9e878e72f64482bdd89f28ac427494-f55670daadbc4a7f88971471629c793f
I20240502 02:02:25.313710  1040 tablet_copy_source_session.cc:205] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 02:02:25.315330  2945 ts_tablet_manager.cc:1875] T f55670daadbc4a7f88971471629c793f P 8b9e878e72f64482bdd89f28ac427494: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 02:02:25.320942  2945 ts_tablet_manager.cc:1888] T f55670daadbc4a7f88971471629c793f P 8b9e878e72f64482bdd89f28ac427494: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 02:02:25.321224  2945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f55670daadbc4a7f88971471629c793f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:25.322574  2927 tablet_bootstrap.cc:492] T 8573fb0d09af421ebbaeb98759e85f4d P eb5a2343514740caa85ff0228a535123: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:02:25.322953  2927 tablet_bootstrap.cc:492] T 8573fb0d09af421ebbaeb98759e85f4d P eb5a2343514740caa85ff0228a535123: Bootstrap complete.
I20240502 02:02:25.323200  2927 ts_tablet_manager.cc:1367] T 8573fb0d09af421ebbaeb98759e85f4d P eb5a2343514740caa85ff0228a535123: Time spent bootstrapping tablet: real 0.037s	user 0.027s	sys 0.008s
I20240502 02:02:25.323421  2945 tablet_copy_client.cc:783] T f55670daadbc4a7f88971471629c793f P 8b9e878e72f64482bdd89f28ac427494: tablet copy: Starting download of 0 data blocks...
I20240502 02:02:25.323657  2945 tablet_copy_client.cc:647] T f55670daadbc4a7f88971471629c793f P 8b9e878e72f64482bdd89f28ac427494: tablet copy: Starting download of 1 WAL segments...
I20240502 02:02:25.324008  2927 raft_consensus.cc:348] T 8573fb0d09af421ebbaeb98759e85f4d P eb5a2343514740caa85ff0228a535123 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } }
I20240502 02:02:25.324247  2927 raft_consensus.cc:729] T 8573fb0d09af421ebbaeb98759e85f4d P eb5a2343514740caa85ff0228a535123 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb5a2343514740caa85ff0228a535123, State: Initialized, Role: LEARNER
I20240502 02:02:25.324473  2927 consensus_queue.cc:260] T 8573fb0d09af421ebbaeb98759e85f4d P eb5a2343514740caa85ff0228a535123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } }
I20240502 02:02:25.325397  2927 ts_tablet_manager.cc:1398] T 8573fb0d09af421ebbaeb98759e85f4d P eb5a2343514740caa85ff0228a535123: Time spent starting tablet: real 0.002s	user 0.005s	sys 0.000s
I20240502 02:02:25.325618  2945 tablet_copy_client.cc:523] T f55670daadbc4a7f88971471629c793f P 8b9e878e72f64482bdd89f28ac427494: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:02:25.326189  1176 tablet_copy_service.cc:342] P 8b9e878e72f64482bdd89f28ac427494: Request end of tablet copy session eb5a2343514740caa85ff0228a535123-8573fb0d09af421ebbaeb98759e85f4d received from {username='slave'} at 127.25.109.132:52325
I20240502 02:02:25.326306  1176 tablet_copy_service.cc:433] P 8b9e878e72f64482bdd89f28ac427494: ending tablet copy session eb5a2343514740caa85ff0228a535123-8573fb0d09af421ebbaeb98759e85f4d on tablet 8573fb0d09af421ebbaeb98759e85f4d with peer eb5a2343514740caa85ff0228a535123
I20240502 02:02:25.329005  2945 tablet_bootstrap.cc:492] T f55670daadbc4a7f88971471629c793f P 8b9e878e72f64482bdd89f28ac427494: Bootstrap starting.
I20240502 02:02:25.363113  2945 tablet_bootstrap.cc:492] T f55670daadbc4a7f88971471629c793f P 8b9e878e72f64482bdd89f28ac427494: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:02:25.363521  2945 tablet_bootstrap.cc:492] T f55670daadbc4a7f88971471629c793f P 8b9e878e72f64482bdd89f28ac427494: Bootstrap complete.
I20240502 02:02:25.363790  2945 ts_tablet_manager.cc:1367] T f55670daadbc4a7f88971471629c793f P 8b9e878e72f64482bdd89f28ac427494: Time spent bootstrapping tablet: real 0.035s	user 0.031s	sys 0.000s
I20240502 02:02:25.364542  2945 raft_consensus.cc:348] T f55670daadbc4a7f88971471629c793f P 8b9e878e72f64482bdd89f28ac427494 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: NON_VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: true } }
I20240502 02:02:25.364778  2945 raft_consensus.cc:729] T f55670daadbc4a7f88971471629c793f P 8b9e878e72f64482bdd89f28ac427494 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8b9e878e72f64482bdd89f28ac427494, State: Initialized, Role: LEARNER
I20240502 02:02:25.364956  2945 consensus_queue.cc:260] T f55670daadbc4a7f88971471629c793f P 8b9e878e72f64482bdd89f28ac427494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: NON_VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: true } }
I20240502 02:02:25.365463  2945 ts_tablet_manager.cc:1398] T f55670daadbc4a7f88971471629c793f P 8b9e878e72f64482bdd89f28ac427494: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 02:02:25.366269  1040 tablet_copy_service.cc:342] P d0e4135c07ac4f2389ffe4e7379df71f: Request end of tablet copy session 8b9e878e72f64482bdd89f28ac427494-f55670daadbc4a7f88971471629c793f received from {username='slave'} at 127.25.109.130:37059
I20240502 02:02:25.366436  1040 tablet_copy_service.cc:433] P d0e4135c07ac4f2389ffe4e7379df71f: ending tablet copy session 8b9e878e72f64482bdd89f28ac427494-f55670daadbc4a7f88971471629c793f on tablet f55670daadbc4a7f88971471629c793f with peer 8b9e878e72f64482bdd89f28ac427494
I20240502 02:02:25.532601  2927 ts_tablet_manager.cc:902] T 11210087d9164d2c9c1303e659230192 P eb5a2343514740caa85ff0228a535123: Initiating tablet copy from peer d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129:33963)
I20240502 02:02:25.534526  2927 tablet_copy_client.cc:308] T 11210087d9164d2c9c1303e659230192 P eb5a2343514740caa85ff0228a535123: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.129:33963
I20240502 02:02:25.535230  1040 tablet_copy_service.cc:140] P d0e4135c07ac4f2389ffe4e7379df71f: Received BeginTabletCopySession request for tablet 11210087d9164d2c9c1303e659230192 from peer eb5a2343514740caa85ff0228a535123 ({username='slave'} at 127.25.109.132:40517)
I20240502 02:02:25.535384  1040 tablet_copy_service.cc:161] P d0e4135c07ac4f2389ffe4e7379df71f: Beginning new tablet copy session on tablet 11210087d9164d2c9c1303e659230192 from peer eb5a2343514740caa85ff0228a535123 at {username='slave'} at 127.25.109.132:40517: session id = eb5a2343514740caa85ff0228a535123-11210087d9164d2c9c1303e659230192
I20240502 02:02:25.537763  1040 tablet_copy_source_session.cc:205] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 02:02:25.539206  2927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11210087d9164d2c9c1303e659230192. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:25.544242  2927 tablet_copy_client.cc:783] T 11210087d9164d2c9c1303e659230192 P eb5a2343514740caa85ff0228a535123: tablet copy: Starting download of 0 data blocks...
I20240502 02:02:25.544502  2927 tablet_copy_client.cc:647] T 11210087d9164d2c9c1303e659230192 P eb5a2343514740caa85ff0228a535123: tablet copy: Starting download of 1 WAL segments...
I20240502 02:02:25.546236  2927 tablet_copy_client.cc:523] T 11210087d9164d2c9c1303e659230192 P eb5a2343514740caa85ff0228a535123: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:02:25.548763  2927 tablet_bootstrap.cc:492] T 11210087d9164d2c9c1303e659230192 P eb5a2343514740caa85ff0228a535123: Bootstrap starting.
I20240502 02:02:25.581279  2927 tablet_bootstrap.cc:492] T 11210087d9164d2c9c1303e659230192 P eb5a2343514740caa85ff0228a535123: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:02:25.581693  2927 tablet_bootstrap.cc:492] T 11210087d9164d2c9c1303e659230192 P eb5a2343514740caa85ff0228a535123: Bootstrap complete.
I20240502 02:02:25.581955  2927 ts_tablet_manager.cc:1367] T 11210087d9164d2c9c1303e659230192 P eb5a2343514740caa85ff0228a535123: Time spent bootstrapping tablet: real 0.033s	user 0.028s	sys 0.003s
I20240502 02:02:25.582630  2927 raft_consensus.cc:348] T 11210087d9164d2c9c1303e659230192 P eb5a2343514740caa85ff0228a535123 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } }
I20240502 02:02:25.582863  2927 raft_consensus.cc:729] T 11210087d9164d2c9c1303e659230192 P eb5a2343514740caa85ff0228a535123 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb5a2343514740caa85ff0228a535123, State: Initialized, Role: LEARNER
I20240502 02:02:25.583087  2927 consensus_queue.cc:260] T 11210087d9164d2c9c1303e659230192 P eb5a2343514740caa85ff0228a535123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: NON_VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: true } }
I20240502 02:02:25.583551  2927 ts_tablet_manager.cc:1398] T 11210087d9164d2c9c1303e659230192 P eb5a2343514740caa85ff0228a535123: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 02:02:25.584379  1040 tablet_copy_service.cc:342] P d0e4135c07ac4f2389ffe4e7379df71f: Request end of tablet copy session eb5a2343514740caa85ff0228a535123-11210087d9164d2c9c1303e659230192 received from {username='slave'} at 127.25.109.132:40517
I20240502 02:02:25.584508  1040 tablet_copy_service.cc:433] P d0e4135c07ac4f2389ffe4e7379df71f: ending tablet copy session eb5a2343514740caa85ff0228a535123-11210087d9164d2c9c1303e659230192 on tablet 11210087d9164d2c9c1303e659230192 with peer eb5a2343514740caa85ff0228a535123
I20240502 02:02:25.754282  1824 raft_consensus.cc:1203] T 8573fb0d09af421ebbaeb98759e85f4d P eb5a2343514740caa85ff0228a535123 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 02:02:25.805537  1155 raft_consensus.cc:1203] T f55670daadbc4a7f88971471629c793f P 8b9e878e72f64482bdd89f28ac427494 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240502 02:02:25.957607  1824 raft_consensus.cc:1203] T 11210087d9164d2c9c1303e659230192 P eb5a2343514740caa85ff0228a535123 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 02:02:26.182744  2995 raft_consensus.cc:1050] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f: attempting to promote NON_VOTER 8b9e878e72f64482bdd89f28ac427494 to VOTER
I20240502 02:02:26.183483  2995 consensus_queue.cc:237] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: false } }
I20240502 02:02:26.185952  1289 raft_consensus.cc:1261] T f55670daadbc4a7f88971471629c793f P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 02:02:26.186020  1824 raft_consensus.cc:1261] T f55670daadbc4a7f88971471629c793f P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 02:02:26.186141  1155 raft_consensus.cc:1261] T f55670daadbc4a7f88971471629c793f P 8b9e878e72f64482bdd89f28ac427494 [term 1 LEARNER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 02:02:26.186635  3013 consensus_queue.cc:1035] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 02:02:26.186996  2995 consensus_queue.cc:1035] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 02:02:26.187265  2838 consensus_queue.cc:1035] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 02:02:26.190487  3013 raft_consensus.cc:2932] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: false } } }
I20240502 02:02:26.191486  1289 raft_consensus.cc:2932] T f55670daadbc4a7f88971471629c793f P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: false } } }
I20240502 02:02:26.191599  1155 raft_consensus.cc:2932] T f55670daadbc4a7f88971471629c793f P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: false } } }
I20240502 02:02:26.195760  1824 raft_consensus.cc:2932] T f55670daadbc4a7f88971471629c793f P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: false } } }
I20240502 02:02:26.196177   867 catalog_manager.cc:5484] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f reported cstate change: config changed from index 8 to 9, 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 02:02:26.204165  1018 consensus_queue.cc:237] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: false } }
I20240502 02:02:26.205919  1824 raft_consensus.cc:1261] T f55670daadbc4a7f88971471629c793f P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 02:02:26.205950  1155 raft_consensus.cc:1261] T f55670daadbc4a7f88971471629c793f P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 02:02:26.206526  3013 consensus_queue.cc:1035] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 02:02:26.206827  2995 consensus_queue.cc:1035] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 02:02:26.209656  2977 raft_consensus.cc:2932] T f55670daadbc4a7f88971471629c793f P d0e4135c07ac4f2389ffe4e7379df71f [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d37e50431f30463c872c5d819ddf2c25 (127.25.109.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: false } } }
I20240502 02:02:26.210280  1824 raft_consensus.cc:2932] T f55670daadbc4a7f88971471629c793f P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d37e50431f30463c872c5d819ddf2c25 (127.25.109.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: false } } }
I20240502 02:02:26.212778   860 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f55670daadbc4a7f88971471629c793f with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 02:02:26.213040  1155 raft_consensus.cc:2932] T f55670daadbc4a7f88971471629c793f P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d37e50431f30463c872c5d819ddf2c25 (127.25.109.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: false } } }
I20240502 02:02:26.214776   866 catalog_manager.cc:5484] T f55670daadbc4a7f88971471629c793f P eb5a2343514740caa85ff0228a535123 reported cstate change: config changed from index 9 to 10, VOTER d37e50431f30463c872c5d819ddf2c25 (127.25.109.131) evicted. New cstate: current_term: 1 leader_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } attrs { promote: false } } }
I20240502 02:02:26.219151  1269 tablet_service.cc:1514] Processing DeleteTablet for tablet f55670daadbc4a7f88971471629c793f with delete_type TABLET_DATA_TOMBSTONED (TS d37e50431f30463c872c5d819ddf2c25 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:55198
I20240502 02:02:26.219784  3022 tablet_replica.cc:330] T f55670daadbc4a7f88971471629c793f P d37e50431f30463c872c5d819ddf2c25: stopping tablet replica
I20240502 02:02:26.220110  3022 raft_consensus.cc:2229] T f55670daadbc4a7f88971471629c793f P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 02:02:26.220630  3022 raft_consensus.cc:2258] T f55670daadbc4a7f88971471629c793f P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 02:02:26.222571  3022 ts_tablet_manager.cc:1875] T f55670daadbc4a7f88971471629c793f P d37e50431f30463c872c5d819ddf2c25: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 02:02:26.228533  3022 ts_tablet_manager.cc:1888] T f55670daadbc4a7f88971471629c793f P d37e50431f30463c872c5d819ddf2c25: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 02:02:26.228796  3022 log.cc:1196] T f55670daadbc4a7f88971471629c793f P d37e50431f30463c872c5d819ddf2c25: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/wals/f55670daadbc4a7f88971471629c793f
I20240502 02:02:26.230172   859 catalog_manager.cc:4830] TS d37e50431f30463c872c5d819ddf2c25 (127.25.109.131:37799): tablet f55670daadbc4a7f88971471629c793f (table rebalance_test_table_1 [id=8991230805234f99bb3820fc5a537d77]) successfully deleted
I20240502 02:02:26.324065  3038 raft_consensus.cc:1050] T 8573fb0d09af421ebbaeb98759e85f4d P 8b9e878e72f64482bdd89f28ac427494: attempting to promote NON_VOTER eb5a2343514740caa85ff0228a535123 to VOTER
I20240502 02:02:26.324795  3038 consensus_queue.cc:237] T 8573fb0d09af421ebbaeb98759e85f4d P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } }
I20240502 02:02:26.327075  1824 raft_consensus.cc:1261] T 8573fb0d09af421ebbaeb98759e85f4d P eb5a2343514740caa85ff0228a535123 [term 1 LEARNER]: Refusing update from remote peer 8b9e878e72f64482bdd89f28ac427494: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:02:26.327261  1289 raft_consensus.cc:1261] T 8573fb0d09af421ebbaeb98759e85f4d P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Refusing update from remote peer 8b9e878e72f64482bdd89f28ac427494: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:02:26.327689  1018 raft_consensus.cc:1261] T 8573fb0d09af421ebbaeb98759e85f4d P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Refusing update from remote peer 8b9e878e72f64482bdd89f28ac427494: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:02:26.327734  3040 consensus_queue.cc:1035] T 8573fb0d09af421ebbaeb98759e85f4d P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:02:26.328167  2979 consensus_queue.cc:1035] T 8573fb0d09af421ebbaeb98759e85f4d P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:02:26.328549  3038 consensus_queue.cc:1035] T 8573fb0d09af421ebbaeb98759e85f4d P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:02:26.331365  3038 raft_consensus.cc:2932] T 8573fb0d09af421ebbaeb98759e85f4d P 8b9e878e72f64482bdd89f28ac427494 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, eb5a2343514740caa85ff0228a535123 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:26.332288  1289 raft_consensus.cc:2932] T 8573fb0d09af421ebbaeb98759e85f4d P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, eb5a2343514740caa85ff0228a535123 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:26.332950  1018 raft_consensus.cc:2932] T 8573fb0d09af421ebbaeb98759e85f4d P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, eb5a2343514740caa85ff0228a535123 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:26.335938  1824 raft_consensus.cc:2932] T 8573fb0d09af421ebbaeb98759e85f4d P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, eb5a2343514740caa85ff0228a535123 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:26.336939   867 catalog_manager.cc:5484] T 8573fb0d09af421ebbaeb98759e85f4d P d37e50431f30463c872c5d819ddf2c25 reported cstate change: config changed from index 5 to 6, eb5a2343514740caa85ff0228a535123 (127.25.109.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8b9e878e72f64482bdd89f28ac427494" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:26.342240  1155 consensus_queue.cc:237] T 8573fb0d09af421ebbaeb98759e85f4d P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } }
I20240502 02:02:26.344194  1018 raft_consensus.cc:1261] T 8573fb0d09af421ebbaeb98759e85f4d P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Refusing update from remote peer 8b9e878e72f64482bdd89f28ac427494: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 02:02:26.344194  1824 raft_consensus.cc:1261] T 8573fb0d09af421ebbaeb98759e85f4d P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Refusing update from remote peer 8b9e878e72f64482bdd89f28ac427494: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 02:02:26.344827  3038 consensus_queue.cc:1035] T 8573fb0d09af421ebbaeb98759e85f4d P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 02:02:26.345191  2979 consensus_queue.cc:1035] T 8573fb0d09af421ebbaeb98759e85f4d P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 02:02:26.347349  3040 raft_consensus.cc:2932] T 8573fb0d09af421ebbaeb98759e85f4d P 8b9e878e72f64482bdd89f28ac427494 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d37e50431f30463c872c5d819ddf2c25 (127.25.109.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:26.348067  1018 raft_consensus.cc:2932] T 8573fb0d09af421ebbaeb98759e85f4d P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d37e50431f30463c872c5d819ddf2c25 (127.25.109.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:26.348701  1824 raft_consensus.cc:2932] T 8573fb0d09af421ebbaeb98759e85f4d P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d37e50431f30463c872c5d819ddf2c25 (127.25.109.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:26.351352   861 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8573fb0d09af421ebbaeb98759e85f4d with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 02:02:26.353773   866 catalog_manager.cc:5484] T 8573fb0d09af421ebbaeb98759e85f4d P d0e4135c07ac4f2389ffe4e7379df71f reported cstate change: config changed from index 6 to 7, VOTER d37e50431f30463c872c5d819ddf2c25 (127.25.109.131) evicted. New cstate: current_term: 1 leader_uuid: "8b9e878e72f64482bdd89f28ac427494" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:26.357964  1269 tablet_service.cc:1514] Processing DeleteTablet for tablet 8573fb0d09af421ebbaeb98759e85f4d with delete_type TABLET_DATA_TOMBSTONED (TS d37e50431f30463c872c5d819ddf2c25 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:55198
I20240502 02:02:26.358381  3022 tablet_replica.cc:330] T 8573fb0d09af421ebbaeb98759e85f4d P d37e50431f30463c872c5d819ddf2c25: stopping tablet replica
I20240502 02:02:26.358611  3022 raft_consensus.cc:2229] T 8573fb0d09af421ebbaeb98759e85f4d P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 02:02:26.359025  3022 raft_consensus.cc:2258] T 8573fb0d09af421ebbaeb98759e85f4d P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 02:02:26.360021  3022 ts_tablet_manager.cc:1875] T 8573fb0d09af421ebbaeb98759e85f4d P d37e50431f30463c872c5d819ddf2c25: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 02:02:26.364637  3022 ts_tablet_manager.cc:1888] T 8573fb0d09af421ebbaeb98759e85f4d P d37e50431f30463c872c5d819ddf2c25: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 02:02:26.364753  3022 log.cc:1196] T 8573fb0d09af421ebbaeb98759e85f4d P d37e50431f30463c872c5d819ddf2c25: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/wals/8573fb0d09af421ebbaeb98759e85f4d
I20240502 02:02:26.365521   859 catalog_manager.cc:4830] TS d37e50431f30463c872c5d819ddf2c25 (127.25.109.131:37799): tablet 8573fb0d09af421ebbaeb98759e85f4d (table rebalance_test_table_1 [id=8991230805234f99bb3820fc5a537d77]) successfully deleted
I20240502 02:02:26.457895  1155 consensus_queue.cc:237] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } }
I20240502 02:02:26.460242  1824 raft_consensus.cc:1261] T 9e7759be006b4a8186aca434c48949b2 P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Refusing update from remote peer 8b9e878e72f64482bdd89f28ac427494: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 02:02:26.460238  1289 raft_consensus.cc:1261] T 9e7759be006b4a8186aca434c48949b2 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Refusing update from remote peer 8b9e878e72f64482bdd89f28ac427494: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 02:02:26.460844  3040 consensus_queue.cc:1035] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 02:02:26.461186  3044 consensus_queue.cc:1035] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 02:02:26.464186  3044 raft_consensus.cc:2932] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:26.464741  1289 raft_consensus.cc:2932] T 9e7759be006b4a8186aca434c48949b2 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:26.464983  1824 raft_consensus.cc:2932] T 9e7759be006b4a8186aca434c48949b2 P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:26.469228   868 catalog_manager.cc:5484] T 9e7759be006b4a8186aca434c48949b2 P d37e50431f30463c872c5d819ddf2c25 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "8b9e878e72f64482bdd89f28ac427494" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:26.475483  1155 consensus_queue.cc:237] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: NON_VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: true } }
I20240502 02:02:26.478260  1824 raft_consensus.cc:1261] T 9e7759be006b4a8186aca434c48949b2 P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Refusing update from remote peer 8b9e878e72f64482bdd89f28ac427494: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 02:02:26.478466  1087 consensus_peers.cc:485] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 -> Peer d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129:33963): Couldn't send request to peer d0e4135c07ac4f2389ffe4e7379df71f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:02:26.478900  3040 consensus_queue.cc:1035] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 02:02:26.479826  1289 raft_consensus.cc:1261] T 9e7759be006b4a8186aca434c48949b2 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Refusing update from remote peer 8b9e878e72f64482bdd89f28ac427494: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 02:02:26.480578  3044 consensus_queue.cc:1035] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 02:02:26.481462  2979 raft_consensus.cc:2932] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: NON_VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: true } } }
I20240502 02:02:26.482668  1824 raft_consensus.cc:2932] T 9e7759be006b4a8186aca434c48949b2 P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: NON_VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: true } } }
I20240502 02:02:26.485671   861 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9e7759be006b4a8186aca434c48949b2 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 02:02:26.487654   867 catalog_manager.cc:5484] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 reported cstate change: config changed from index 7 to 8, NON_VOTER d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129) added. New cstate: current_term: 1 leader_uuid: "8b9e878e72f64482bdd89f28ac427494" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: NON_VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 02:02:26.487660  1289 raft_consensus.cc:2932] T 9e7759be006b4a8186aca434c48949b2 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: NON_VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: true } } }
I20240502 02:02:26.509136  2995 raft_consensus.cc:1050] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f: attempting to promote NON_VOTER eb5a2343514740caa85ff0228a535123 to VOTER
I20240502 02:02:26.510022  2995 consensus_queue.cc:237] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } }
I20240502 02:02:26.512243  1155 raft_consensus.cc:1261] T 11210087d9164d2c9c1303e659230192 P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:02:26.512348  1289 raft_consensus.cc:1261] T 11210087d9164d2c9c1303e659230192 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:02:26.512559  1824 raft_consensus.cc:1261] T 11210087d9164d2c9c1303e659230192 P eb5a2343514740caa85ff0228a535123 [term 1 LEARNER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 02:02:26.512842  2995 consensus_queue.cc:1035] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:02:26.513161  3013 consensus_queue.cc:1035] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:02:26.513496  2977 consensus_queue.cc:1035] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 02:02:26.516902  2995 raft_consensus.cc:2932] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, eb5a2343514740caa85ff0228a535123 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:26.518013  1824 raft_consensus.cc:2932] T 11210087d9164d2c9c1303e659230192 P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, eb5a2343514740caa85ff0228a535123 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:26.518077  1289 raft_consensus.cc:2932] T 11210087d9164d2c9c1303e659230192 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, eb5a2343514740caa85ff0228a535123 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:26.520967  1155 raft_consensus.cc:2932] T 11210087d9164d2c9c1303e659230192 P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, eb5a2343514740caa85ff0228a535123 (127.25.109.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:26.522854   867 catalog_manager.cc:5484] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f reported cstate change: config changed from index 5 to 6, eb5a2343514740caa85ff0228a535123 (127.25.109.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 02:02:26.528066  1018 consensus_queue.cc:237] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } }
I20240502 02:02:26.529917  1824 raft_consensus.cc:1261] T 11210087d9164d2c9c1303e659230192 P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 02:02:26.529982  1155 raft_consensus.cc:1261] T 11210087d9164d2c9c1303e659230192 P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 02:02:26.530552  2995 consensus_queue.cc:1035] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 02:02:26.530880  2977 consensus_queue.cc:1035] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 02:02:26.533169  3013 raft_consensus.cc:2932] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d37e50431f30463c872c5d819ddf2c25 (127.25.109.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:26.533861  1155 raft_consensus.cc:2932] T 11210087d9164d2c9c1303e659230192 P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d37e50431f30463c872c5d819ddf2c25 (127.25.109.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:26.534032  1824 raft_consensus.cc:2932] T 11210087d9164d2c9c1303e659230192 P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d37e50431f30463c872c5d819ddf2c25 (127.25.109.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:26.536144   860 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 11210087d9164d2c9c1303e659230192 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 02:02:26.537803   866 catalog_manager.cc:5484] T 11210087d9164d2c9c1303e659230192 P d0e4135c07ac4f2389ffe4e7379df71f reported cstate change: config changed from index 6 to 7, VOTER d37e50431f30463c872c5d819ddf2c25 (127.25.109.131) evicted. New cstate: current_term: 1 leader_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 02:02:26.542009  1269 tablet_service.cc:1514] Processing DeleteTablet for tablet 11210087d9164d2c9c1303e659230192 with delete_type TABLET_DATA_TOMBSTONED (TS d37e50431f30463c872c5d819ddf2c25 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:55198
I20240502 02:02:26.542416  3022 tablet_replica.cc:330] T 11210087d9164d2c9c1303e659230192 P d37e50431f30463c872c5d819ddf2c25: stopping tablet replica
I20240502 02:02:26.542618  3022 raft_consensus.cc:2229] T 11210087d9164d2c9c1303e659230192 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 02:02:26.543006  3022 raft_consensus.cc:2258] T 11210087d9164d2c9c1303e659230192 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 02:02:26.543916  3022 ts_tablet_manager.cc:1875] T 11210087d9164d2c9c1303e659230192 P d37e50431f30463c872c5d819ddf2c25: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 02:02:26.548241  3022 ts_tablet_manager.cc:1888] T 11210087d9164d2c9c1303e659230192 P d37e50431f30463c872c5d819ddf2c25: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 02:02:26.548360  3022 log.cc:1196] T 11210087d9164d2c9c1303e659230192 P d37e50431f30463c872c5d819ddf2c25: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/wals/11210087d9164d2c9c1303e659230192
I20240502 02:02:26.549019   859 catalog_manager.cc:4830] TS d37e50431f30463c872c5d819ddf2c25 (127.25.109.131:37799): tablet 11210087d9164d2c9c1303e659230192 (table rebalance_test_table_2 [id=57c9d0a8579e4e908683ee50f24d5aa1]) successfully deleted
I20240502 02:02:26.721274  1289 consensus_queue.cc:237] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } }
I20240502 02:02:26.723416  1824 raft_consensus.cc:1261] T b1005c925b954ca39a9ee6f2677b3a1f P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Refusing update from remote peer d37e50431f30463c872c5d819ddf2c25: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 02:02:26.723500  1155 raft_consensus.cc:1261] T b1005c925b954ca39a9ee6f2677b3a1f P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Refusing update from remote peer d37e50431f30463c872c5d819ddf2c25: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 02:02:26.724012  3080 consensus_queue.cc:1035] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 02:02:26.724380  2567 consensus_queue.cc:1035] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 02:02:26.727264  3080 raft_consensus.cc:2932] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:26.728111  1824 raft_consensus.cc:2932] T b1005c925b954ca39a9ee6f2677b3a1f P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:26.733500   868 catalog_manager.cc:5484] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "d37e50431f30463c872c5d819ddf2c25" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 02:02:26.740516  1289 consensus_queue.cc:237] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: NON_VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: true } }
I20240502 02:02:26.745170  1824 raft_consensus.cc:1261] T b1005c925b954ca39a9ee6f2677b3a1f P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Refusing update from remote peer d37e50431f30463c872c5d819ddf2c25: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 02:02:26.745383  1155 raft_consensus.cc:1261] T b1005c925b954ca39a9ee6f2677b3a1f P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Refusing update from remote peer d37e50431f30463c872c5d819ddf2c25: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 02:02:26.745842  2567 consensus_queue.cc:1035] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 02:02:26.748018  2567 consensus_queue.cc:1035] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 02:02:26.748605  1221 consensus_peers.cc:485] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 -> Peer d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129:33963): Couldn't send request to peer d0e4135c07ac4f2389ffe4e7379df71f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:02:26.749418  1155 raft_consensus.cc:2932] T b1005c925b954ca39a9ee6f2677b3a1f P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:26.749540  2567 raft_consensus.cc:2932] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: NON_VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: true } } }
I20240502 02:02:26.751909  1824 raft_consensus.cc:2932] T b1005c925b954ca39a9ee6f2677b3a1f P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: NON_VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: true } } }
I20240502 02:02:26.756227  1155 raft_consensus.cc:2932] T b1005c925b954ca39a9ee6f2677b3a1f P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: NON_VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: true } } }
I20240502 02:02:26.756906   868 catalog_manager.cc:5484] T b1005c925b954ca39a9ee6f2677b3a1f P eb5a2343514740caa85ff0228a535123 reported cstate change: config changed from index 7 to 8, NON_VOTER d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129) added. New cstate: current_term: 1 leader_uuid: "d37e50431f30463c872c5d819ddf2c25" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: NON_VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: true } } }
I20240502 02:02:26.760937   859 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b1005c925b954ca39a9ee6f2677b3a1f with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 02:02:26.992516  1288 consensus_queue.cc:237] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } }
I20240502 02:02:26.994848  1824 raft_consensus.cc:1261] T 884b0248d66f4575a7e8e3ead42c58c6 P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Refusing update from remote peer d37e50431f30463c872c5d819ddf2c25: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 02:02:26.994850  1156 raft_consensus.cc:1261] T 884b0248d66f4575a7e8e3ead42c58c6 P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Refusing update from remote peer d37e50431f30463c872c5d819ddf2c25: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 02:02:26.995414  3080 consensus_queue.cc:1035] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 02:02:26.995718  2567 consensus_queue.cc:1035] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 02:02:26.998955  3089 raft_consensus.cc:2932] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:26.999321  1824 raft_consensus.cc:2932] T 884b0248d66f4575a7e8e3ead42c58c6 P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:27.004060   873 catalog_manager.cc:5484] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "d37e50431f30463c872c5d819ddf2c25" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 02:02:27.004827  1155 raft_consensus.cc:2932] T 884b0248d66f4575a7e8e3ead42c58c6 P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } }
I20240502 02:02:27.009482  1288 consensus_queue.cc:237] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: NON_VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: true } }
I20240502 02:02:27.012948  1155 raft_consensus.cc:1261] T 884b0248d66f4575a7e8e3ead42c58c6 P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Refusing update from remote peer d37e50431f30463c872c5d819ddf2c25: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 02:02:27.012929  1221 consensus_peers.cc:485] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 -> Peer d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129:33963): Couldn't send request to peer d0e4135c07ac4f2389ffe4e7379df71f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 02:02:27.013731  3089 consensus_queue.cc:1035] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 02:02:27.016227  1824 raft_consensus.cc:1261] T 884b0248d66f4575a7e8e3ead42c58c6 P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Refusing update from remote peer d37e50431f30463c872c5d819ddf2c25: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 02:02:27.016497  2567 raft_consensus.cc:2932] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: NON_VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: true } } }
I20240502 02:02:27.016798  3089 consensus_queue.cc:1035] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 02:02:27.018338  1824 raft_consensus.cc:2932] T 884b0248d66f4575a7e8e3ead42c58c6 P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: NON_VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: true } } }
I20240502 02:02:27.020781   859 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 884b0248d66f4575a7e8e3ead42c58c6 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 02:02:27.020565  1155 raft_consensus.cc:2932] T 884b0248d66f4575a7e8e3ead42c58c6 P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: NON_VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: true } } }
I20240502 02:02:27.022071   873 catalog_manager.cc:5484] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 reported cstate change: config changed from index 7 to 8, NON_VOTER d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129) added. New cstate: current_term: 1 leader_uuid: "d37e50431f30463c872c5d819ddf2c25" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: NON_VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 02:02:27.071141  3125 ts_tablet_manager.cc:902] T 9e7759be006b4a8186aca434c48949b2 P d0e4135c07ac4f2389ffe4e7379df71f: Initiating tablet copy from peer 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130:43631)
I20240502 02:02:27.072705  3125 tablet_copy_client.cc:308] T 9e7759be006b4a8186aca434c48949b2 P d0e4135c07ac4f2389ffe4e7379df71f: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.130:43631
I20240502 02:02:27.073506  1176 tablet_copy_service.cc:140] P 8b9e878e72f64482bdd89f28ac427494: Received BeginTabletCopySession request for tablet 9e7759be006b4a8186aca434c48949b2 from peer d0e4135c07ac4f2389ffe4e7379df71f ({username='slave'} at 127.25.109.129:51057)
I20240502 02:02:27.073673  1176 tablet_copy_service.cc:161] P 8b9e878e72f64482bdd89f28ac427494: Beginning new tablet copy session on tablet 9e7759be006b4a8186aca434c48949b2 from peer d0e4135c07ac4f2389ffe4e7379df71f at {username='slave'} at 127.25.109.129:51057: session id = d0e4135c07ac4f2389ffe4e7379df71f-9e7759be006b4a8186aca434c48949b2
I20240502 02:02:27.075657  1176 tablet_copy_source_session.cc:205] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 02:02:27.077008  3125 ts_tablet_manager.cc:1875] T 9e7759be006b4a8186aca434c48949b2 P d0e4135c07ac4f2389ffe4e7379df71f: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 02:02:27.081872  3125 ts_tablet_manager.cc:1888] T 9e7759be006b4a8186aca434c48949b2 P d0e4135c07ac4f2389ffe4e7379df71f: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 02:02:27.082127  3125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e7759be006b4a8186aca434c48949b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:27.084565  3125 tablet_copy_client.cc:783] T 9e7759be006b4a8186aca434c48949b2 P d0e4135c07ac4f2389ffe4e7379df71f: tablet copy: Starting download of 0 data blocks...
I20240502 02:02:27.084851  3125 tablet_copy_client.cc:647] T 9e7759be006b4a8186aca434c48949b2 P d0e4135c07ac4f2389ffe4e7379df71f: tablet copy: Starting download of 1 WAL segments...
I20240502 02:02:27.086707  3125 tablet_copy_client.cc:523] T 9e7759be006b4a8186aca434c48949b2 P d0e4135c07ac4f2389ffe4e7379df71f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:02:27.089624  3125 tablet_bootstrap.cc:492] T 9e7759be006b4a8186aca434c48949b2 P d0e4135c07ac4f2389ffe4e7379df71f: Bootstrap starting.
I20240502 02:02:27.123344  3125 tablet_bootstrap.cc:492] T 9e7759be006b4a8186aca434c48949b2 P d0e4135c07ac4f2389ffe4e7379df71f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:02:27.123752  3125 tablet_bootstrap.cc:492] T 9e7759be006b4a8186aca434c48949b2 P d0e4135c07ac4f2389ffe4e7379df71f: Bootstrap complete.
I20240502 02:02:27.124015  3125 ts_tablet_manager.cc:1367] T 9e7759be006b4a8186aca434c48949b2 P d0e4135c07ac4f2389ffe4e7379df71f: Time spent bootstrapping tablet: real 0.034s	user 0.032s	sys 0.004s
I20240502 02:02:27.124702  3125 raft_consensus.cc:348] T 9e7759be006b4a8186aca434c48949b2 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: NON_VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: true } }
I20240502 02:02:27.124929  3125 raft_consensus.cc:729] T 9e7759be006b4a8186aca434c48949b2 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d0e4135c07ac4f2389ffe4e7379df71f, State: Initialized, Role: LEARNER
I20240502 02:02:27.125099  3125 consensus_queue.cc:260] T 9e7759be006b4a8186aca434c48949b2 P d0e4135c07ac4f2389ffe4e7379df71f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: NON_VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: true } }
I20240502 02:02:27.125595  3125 ts_tablet_manager.cc:1398] T 9e7759be006b4a8186aca434c48949b2 P d0e4135c07ac4f2389ffe4e7379df71f: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 02:02:27.126557  1176 tablet_copy_service.cc:342] P 8b9e878e72f64482bdd89f28ac427494: Request end of tablet copy session d0e4135c07ac4f2389ffe4e7379df71f-9e7759be006b4a8186aca434c48949b2 received from {username='slave'} at 127.25.109.129:51057
I20240502 02:02:27.126708  1176 tablet_copy_service.cc:433] P 8b9e878e72f64482bdd89f28ac427494: ending tablet copy session d0e4135c07ac4f2389ffe4e7379df71f-9e7759be006b4a8186aca434c48949b2 on tablet 9e7759be006b4a8186aca434c48949b2 with peer d0e4135c07ac4f2389ffe4e7379df71f
I20240502 02:02:27.296205  3125 ts_tablet_manager.cc:902] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f: Initiating tablet copy from peer d37e50431f30463c872c5d819ddf2c25 (127.25.109.131:37799)
I20240502 02:02:27.297425  3125 tablet_copy_client.cc:308] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.131:37799
I20240502 02:02:27.298135  1309 tablet_copy_service.cc:140] P d37e50431f30463c872c5d819ddf2c25: Received BeginTabletCopySession request for tablet b1005c925b954ca39a9ee6f2677b3a1f from peer d0e4135c07ac4f2389ffe4e7379df71f ({username='slave'} at 127.25.109.129:48755)
I20240502 02:02:27.298296  1309 tablet_copy_service.cc:161] P d37e50431f30463c872c5d819ddf2c25: Beginning new tablet copy session on tablet b1005c925b954ca39a9ee6f2677b3a1f from peer d0e4135c07ac4f2389ffe4e7379df71f at {username='slave'} at 127.25.109.129:48755: session id = d0e4135c07ac4f2389ffe4e7379df71f-b1005c925b954ca39a9ee6f2677b3a1f
I20240502 02:02:27.300380  1309 tablet_copy_source_session.cc:205] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 02:02:27.301499  3125 ts_tablet_manager.cc:1875] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 02:02:27.306140  3125 ts_tablet_manager.cc:1888] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 02:02:27.306389  3125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1005c925b954ca39a9ee6f2677b3a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:27.308563  3125 tablet_copy_client.cc:783] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f: tablet copy: Starting download of 0 data blocks...
I20240502 02:02:27.308782  3125 tablet_copy_client.cc:647] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f: tablet copy: Starting download of 1 WAL segments...
I20240502 02:02:27.310439  3125 tablet_copy_client.cc:523] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:02:27.312911  3125 tablet_bootstrap.cc:492] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f: Bootstrap starting.
I20240502 02:02:27.345223  3125 tablet_bootstrap.cc:492] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:02:27.345645  3125 tablet_bootstrap.cc:492] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f: Bootstrap complete.
I20240502 02:02:27.345888  3125 ts_tablet_manager.cc:1367] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f: Time spent bootstrapping tablet: real 0.033s	user 0.028s	sys 0.004s
I20240502 02:02:27.346554  3125 raft_consensus.cc:348] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: NON_VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: true } }
I20240502 02:02:27.346779  3125 raft_consensus.cc:729] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d0e4135c07ac4f2389ffe4e7379df71f, State: Initialized, Role: LEARNER
I20240502 02:02:27.346980  3125 consensus_queue.cc:260] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: NON_VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: true } }
I20240502 02:02:27.347496  3125 ts_tablet_manager.cc:1398] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 02:02:27.348423  1309 tablet_copy_service.cc:342] P d37e50431f30463c872c5d819ddf2c25: Request end of tablet copy session d0e4135c07ac4f2389ffe4e7379df71f-b1005c925b954ca39a9ee6f2677b3a1f received from {username='slave'} at 127.25.109.129:48755
I20240502 02:02:27.348603  1309 tablet_copy_service.cc:433] P d37e50431f30463c872c5d819ddf2c25: ending tablet copy session d0e4135c07ac4f2389ffe4e7379df71f-b1005c925b954ca39a9ee6f2677b3a1f on tablet b1005c925b954ca39a9ee6f2677b3a1f with peer d0e4135c07ac4f2389ffe4e7379df71f
I20240502 02:02:27.622045  3125 ts_tablet_manager.cc:902] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f: Initiating tablet copy from peer d37e50431f30463c872c5d819ddf2c25 (127.25.109.131:37799)
I20240502 02:02:27.623368  3125 tablet_copy_client.cc:308] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f: tablet copy: Beginning tablet copy session from remote peer at address 127.25.109.131:37799
I20240502 02:02:27.624094  1309 tablet_copy_service.cc:140] P d37e50431f30463c872c5d819ddf2c25: Received BeginTabletCopySession request for tablet 884b0248d66f4575a7e8e3ead42c58c6 from peer d0e4135c07ac4f2389ffe4e7379df71f ({username='slave'} at 127.25.109.129:48755)
I20240502 02:02:27.624272  1309 tablet_copy_service.cc:161] P d37e50431f30463c872c5d819ddf2c25: Beginning new tablet copy session on tablet 884b0248d66f4575a7e8e3ead42c58c6 from peer d0e4135c07ac4f2389ffe4e7379df71f at {username='slave'} at 127.25.109.129:48755: session id = d0e4135c07ac4f2389ffe4e7379df71f-884b0248d66f4575a7e8e3ead42c58c6
I20240502 02:02:27.626206  1309 tablet_copy_source_session.cc:205] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 02:02:27.627202  3125 ts_tablet_manager.cc:1875] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 02:02:27.631605  3125 ts_tablet_manager.cc:1888] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 02:02:27.631889  3125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 884b0248d66f4575a7e8e3ead42c58c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 02:02:27.634001  3125 tablet_copy_client.cc:783] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f: tablet copy: Starting download of 0 data blocks...
I20240502 02:02:27.634207  3125 tablet_copy_client.cc:647] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f: tablet copy: Starting download of 1 WAL segments...
I20240502 02:02:27.635707  3125 tablet_copy_client.cc:523] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 02:02:27.638163  3125 tablet_bootstrap.cc:492] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f: Bootstrap starting.
I20240502 02:02:27.667690  3125 tablet_bootstrap.cc:492] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 02:02:27.668128  3125 tablet_bootstrap.cc:492] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f: Bootstrap complete.
I20240502 02:02:27.668396  3125 ts_tablet_manager.cc:1367] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f: Time spent bootstrapping tablet: real 0.030s	user 0.030s	sys 0.002s
I20240502 02:02:27.669042  3125 raft_consensus.cc:348] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: NON_VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: true } }
I20240502 02:02:27.669282  3125 raft_consensus.cc:729] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d0e4135c07ac4f2389ffe4e7379df71f, State: Initialized, Role: LEARNER
I20240502 02:02:27.669471  3125 consensus_queue.cc:260] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: NON_VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: true } }
I20240502 02:02:27.670017  3125 ts_tablet_manager.cc:1398] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 02:02:27.670861  1309 tablet_copy_service.cc:342] P d37e50431f30463c872c5d819ddf2c25: Request end of tablet copy session d0e4135c07ac4f2389ffe4e7379df71f-884b0248d66f4575a7e8e3ead42c58c6 received from {username='slave'} at 127.25.109.129:48755
I20240502 02:02:27.670979  1309 tablet_copy_service.cc:433] P d37e50431f30463c872c5d819ddf2c25: ending tablet copy session d0e4135c07ac4f2389ffe4e7379df71f-884b0248d66f4575a7e8e3ead42c58c6 on tablet 884b0248d66f4575a7e8e3ead42c58c6 with peer d0e4135c07ac4f2389ffe4e7379df71f
I20240502 02:02:27.684903  1018 raft_consensus.cc:1203] T 9e7759be006b4a8186aca434c48949b2 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240502 02:02:27.699388  1018 raft_consensus.cc:1203] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240502 02:02:28.136338  3168 raft_consensus.cc:1050] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25: attempting to promote NON_VOTER d0e4135c07ac4f2389ffe4e7379df71f to VOTER
I20240502 02:02:28.137091  3168 consensus_queue.cc:237] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: false } }
I20240502 02:02:28.139936  1824 raft_consensus.cc:1261] T b1005c925b954ca39a9ee6f2677b3a1f P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Refusing update from remote peer d37e50431f30463c872c5d819ddf2c25: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 02:02:28.140290  1156 raft_consensus.cc:1261] T b1005c925b954ca39a9ee6f2677b3a1f P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Refusing update from remote peer d37e50431f30463c872c5d819ddf2c25: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 02:02:28.140319  1018 raft_consensus.cc:1261] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f [term 1 LEARNER]: Refusing update from remote peer d37e50431f30463c872c5d819ddf2c25: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 02:02:28.140806  2567 consensus_queue.cc:1035] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 02:02:28.141184  3208 consensus_queue.cc:1035] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 02:02:28.141491  3168 consensus_queue.cc:1035] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 02:02:28.144485  2567 raft_consensus.cc:2932] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: false } } }
I20240502 02:02:28.145179  1018 raft_consensus.cc:2932] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: false } } }
I20240502 02:02:28.145735  1156 raft_consensus.cc:2932] T b1005c925b954ca39a9ee6f2677b3a1f P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: false } } }
I20240502 02:02:28.149924  1824 raft_consensus.cc:2932] T b1005c925b954ca39a9ee6f2677b3a1f P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: false } } }
I20240502 02:02:28.150039   868 catalog_manager.cc:5484] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 reported cstate change: config changed from index 8 to 9, d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d37e50431f30463c872c5d819ddf2c25" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 02:02:28.170387  1018 raft_consensus.cc:1203] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240502 02:02:28.196967  1288 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b1005c925b954ca39a9ee6f2677b3a1f"
dest_uuid: "d37e50431f30463c872c5d819ddf2c25"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:38266
I20240502 02:02:28.197222  1288 raft_consensus.cc:595] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [term 1 LEADER]: Received request to transfer leadership
I20240502 02:02:28.210285  3175 raft_consensus.cc:1050] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494: attempting to promote NON_VOTER d0e4135c07ac4f2389ffe4e7379df71f to VOTER
I20240502 02:02:28.210966  3175 consensus_queue.cc:237] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: false } }
I20240502 02:02:28.213071  1018 raft_consensus.cc:1261] T 9e7759be006b4a8186aca434c48949b2 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 LEARNER]: Refusing update from remote peer 8b9e878e72f64482bdd89f28ac427494: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 02:02:28.213513  1824 raft_consensus.cc:1261] T 9e7759be006b4a8186aca434c48949b2 P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Refusing update from remote peer 8b9e878e72f64482bdd89f28ac427494: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 02:02:28.213788  1288 raft_consensus.cc:1261] T 9e7759be006b4a8186aca434c48949b2 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Refusing update from remote peer 8b9e878e72f64482bdd89f28ac427494: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 02:02:28.214134  3240 consensus_queue.cc:1035] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 02:02:28.214466  3238 consensus_queue.cc:1035] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 02:02:28.220957  3240 consensus_queue.cc:1035] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 02:02:28.228639  3175 raft_consensus.cc:2932] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: false } } }
I20240502 02:02:28.232695  1018 raft_consensus.cc:2932] T 9e7759be006b4a8186aca434c48949b2 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: false } } }
I20240502 02:02:28.233127  1824 raft_consensus.cc:2932] T 9e7759be006b4a8186aca434c48949b2 P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: false } } }
I20240502 02:02:28.234100   867 catalog_manager.cc:5484] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 reported cstate change: config changed from index 8 to 9, d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8b9e878e72f64482bdd89f28ac427494" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 02:02:28.236414  1288 raft_consensus.cc:2932] T 9e7759be006b4a8186aca434c48949b2 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: false } } }
I20240502 02:02:28.241099  1155 consensus_queue.cc:237] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: false } }
I20240502 02:02:28.243296  1018 raft_consensus.cc:1261] T 9e7759be006b4a8186aca434c48949b2 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Refusing update from remote peer 8b9e878e72f64482bdd89f28ac427494: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 02:02:28.243476  1824 raft_consensus.cc:1261] T 9e7759be006b4a8186aca434c48949b2 P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Refusing update from remote peer 8b9e878e72f64482bdd89f28ac427494: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 02:02:28.243886  3241 consensus_queue.cc:1035] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 02:02:28.244258  3240 consensus_queue.cc:1035] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 02:02:28.246454  3175 raft_consensus.cc:2932] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d37e50431f30463c872c5d819ddf2c25 (127.25.109.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: false } } }
I20240502 02:02:28.247180  1018 raft_consensus.cc:2932] T 9e7759be006b4a8186aca434c48949b2 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d37e50431f30463c872c5d819ddf2c25 (127.25.109.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: false } } }
I20240502 02:02:28.248096  1824 raft_consensus.cc:2932] T 9e7759be006b4a8186aca434c48949b2 P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d37e50431f30463c872c5d819ddf2c25 (127.25.109.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: false } } }
I20240502 02:02:28.249506   861 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9e7759be006b4a8186aca434c48949b2 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 02:02:28.250893   866 catalog_manager.cc:5484] T 9e7759be006b4a8186aca434c48949b2 P 8b9e878e72f64482bdd89f28ac427494 reported cstate change: config changed from index 9 to 10, VOTER d37e50431f30463c872c5d819ddf2c25 (127.25.109.131) evicted. New cstate: current_term: 1 leader_uuid: "8b9e878e72f64482bdd89f28ac427494" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 02:02:28.257478  1269 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e7759be006b4a8186aca434c48949b2 with delete_type TABLET_DATA_TOMBSTONED (TS d37e50431f30463c872c5d819ddf2c25 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:55198
I20240502 02:02:28.258235  3246 tablet_replica.cc:330] T 9e7759be006b4a8186aca434c48949b2 P d37e50431f30463c872c5d819ddf2c25: stopping tablet replica
I20240502 02:02:28.258699  3246 raft_consensus.cc:2229] T 9e7759be006b4a8186aca434c48949b2 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 02:02:28.259271  3246 raft_consensus.cc:2258] T 9e7759be006b4a8186aca434c48949b2 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 02:02:28.264623  3246 ts_tablet_manager.cc:1875] T 9e7759be006b4a8186aca434c48949b2 P d37e50431f30463c872c5d819ddf2c25: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 02:02:28.269121  3246 ts_tablet_manager.cc:1888] T 9e7759be006b4a8186aca434c48949b2 P d37e50431f30463c872c5d819ddf2c25: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 02:02:28.269240  3246 log.cc:1196] T 9e7759be006b4a8186aca434c48949b2 P d37e50431f30463c872c5d819ddf2c25: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/wals/9e7759be006b4a8186aca434c48949b2
I20240502 02:02:28.270033   859 catalog_manager.cc:4830] TS d37e50431f30463c872c5d819ddf2c25 (127.25.109.131:37799): tablet 9e7759be006b4a8186aca434c48949b2 (table rebalance_test_table_0 [id=3920dc53a6a846338bdd42178520e7af]) successfully deleted
I20240502 02:02:28.433265  1288 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b1005c925b954ca39a9ee6f2677b3a1f"
dest_uuid: "d37e50431f30463c872c5d819ddf2c25"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:38280
I20240502 02:02:28.433537  1288 raft_consensus.cc:595] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [term 1 LEADER]: Received request to transfer leadership
I20240502 02:02:28.565670  3168 raft_consensus.cc:979] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25: : Instructing follower d0e4135c07ac4f2389ffe4e7379df71f to start an election
I20240502 02:02:28.565904  2567 raft_consensus.cc:1067] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [term 1 LEADER]: Signalling peer d0e4135c07ac4f2389ffe4e7379df71f to start an election
I20240502 02:02:28.566720  1018 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b1005c925b954ca39a9ee6f2677b3a1f"
dest_uuid: "d0e4135c07ac4f2389ffe4e7379df71f"
 from {username='slave'} at 127.25.109.131:48671
I20240502 02:02:28.566941  1018 raft_consensus.cc:482] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 02:02:28.567050  1018 raft_consensus.cc:3037] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Advancing to term 2
I20240502 02:02:28.569154  1018 raft_consensus.cc:504] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: false } }
I20240502 02:02:28.570124  1018 leader_election.cc:288] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f [CANDIDATE]: Term 2 election: Requested vote from peers 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130:43631), d37e50431f30463c872c5d819ddf2c25 (127.25.109.131:37799), eb5a2343514740caa85ff0228a535123 (127.25.109.132:44441)
I20240502 02:02:28.570566  1155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1005c925b954ca39a9ee6f2677b3a1f" candidate_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "8b9e878e72f64482bdd89f28ac427494"
I20240502 02:02:28.570648  1288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1005c925b954ca39a9ee6f2677b3a1f" candidate_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "d37e50431f30463c872c5d819ddf2c25"
I20240502 02:02:28.570835  1155 raft_consensus.cc:3037] T b1005c925b954ca39a9ee6f2677b3a1f P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Advancing to term 2
I20240502 02:02:28.570869  1288 raft_consensus.cc:3032] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 02:02:28.570797  1824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1005c925b954ca39a9ee6f2677b3a1f" candidate_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "eb5a2343514740caa85ff0228a535123"
I20240502 02:02:28.570948  1288 raft_consensus.cc:729] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d37e50431f30463c872c5d819ddf2c25, State: Running, Role: LEADER
I20240502 02:02:28.571065  1824 raft_consensus.cc:3037] T b1005c925b954ca39a9ee6f2677b3a1f P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Advancing to term 2
I20240502 02:02:28.571280  1288 consensus_queue.cc:260] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: false } }
I20240502 02:02:28.571657  1288 raft_consensus.cc:3037] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Advancing to term 2
I20240502 02:02:28.573519  1288 raft_consensus.cc:2454] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0e4135c07ac4f2389ffe4e7379df71f in term 2.
I20240502 02:02:28.573681  1155 raft_consensus.cc:2454] T b1005c925b954ca39a9ee6f2677b3a1f P 8b9e878e72f64482bdd89f28ac427494 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0e4135c07ac4f2389ffe4e7379df71f in term 2.
I20240502 02:02:28.573734  1824 raft_consensus.cc:2454] T b1005c925b954ca39a9ee6f2677b3a1f P eb5a2343514740caa85ff0228a535123 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0e4135c07ac4f2389ffe4e7379df71f in term 2.
I20240502 02:02:28.574215   954 leader_election.cc:302] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: d0e4135c07ac4f2389ffe4e7379df71f, d37e50431f30463c872c5d819ddf2c25, eb5a2343514740caa85ff0228a535123; no voters: 
I20240502 02:02:28.574513  2995 raft_consensus.cc:2781] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 02:02:28.574863  2995 raft_consensus.cc:686] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f [term 2 LEADER]: Becoming Leader. State: Replica: d0e4135c07ac4f2389ffe4e7379df71f, State: Running, Role: LEADER
I20240502 02:02:28.575197  2995 consensus_queue.cc:237] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: false } }
I20240502 02:02:28.577751   873 catalog_manager.cc:5484] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f reported cstate change: term changed from 1 to 2, leader changed from d37e50431f30463c872c5d819ddf2c25 (127.25.109.131) to d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129). New cstate: current_term: 2 leader_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 02:02:28.789578  3168 raft_consensus.cc:1050] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25: attempting to promote NON_VOTER d0e4135c07ac4f2389ffe4e7379df71f to VOTER
I20240502 02:02:28.790342  3168 consensus_queue.cc:237] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: false } }
I20240502 02:02:28.792814  1018 raft_consensus.cc:1261] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 LEARNER]: Refusing update from remote peer d37e50431f30463c872c5d819ddf2c25: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 02:02:28.792814  1824 raft_consensus.cc:1261] T 884b0248d66f4575a7e8e3ead42c58c6 P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Refusing update from remote peer d37e50431f30463c872c5d819ddf2c25: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 02:02:28.793377  1155 raft_consensus.cc:1261] T 884b0248d66f4575a7e8e3ead42c58c6 P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Refusing update from remote peer d37e50431f30463c872c5d819ddf2c25: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 02:02:28.793413  3168 consensus_queue.cc:1035] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 02:02:28.793773  2567 consensus_queue.cc:1035] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 02:02:28.794145  3280 consensus_queue.cc:1035] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 02:02:28.797487  2567 raft_consensus.cc:2932] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: false } } }
I20240502 02:02:28.798161  1824 raft_consensus.cc:2932] T 884b0248d66f4575a7e8e3ead42c58c6 P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: false } } }
I20240502 02:02:28.798519  1018 raft_consensus.cc:2932] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: false } } }
I20240502 02:02:28.801785  1155 raft_consensus.cc:2932] T 884b0248d66f4575a7e8e3ead42c58c6 P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: false } } }
I20240502 02:02:28.802834   873 catalog_manager.cc:5484] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 reported cstate change: config changed from index 8 to 9, d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d37e50431f30463c872c5d819ddf2c25" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 02:02:28.880689  1288 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "884b0248d66f4575a7e8e3ead42c58c6"
dest_uuid: "d37e50431f30463c872c5d819ddf2c25"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:38290
I20240502 02:02:28.880941  1288 raft_consensus.cc:595] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [term 1 LEADER]: Received request to transfer leadership
I20240502 02:02:28.957185  1155 raft_consensus.cc:1261] T b1005c925b954ca39a9ee6f2677b3a1f P 8b9e878e72f64482bdd89f28ac427494 [term 2 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 02:02:28.957794  2995 consensus_queue.cc:1035] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 02:02:29.069520  1824 raft_consensus.cc:1261] T b1005c925b954ca39a9ee6f2677b3a1f P eb5a2343514740caa85ff0228a535123 [term 2 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 02:02:29.070170  2995 consensus_queue.cc:1035] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 02:02:29.075248  1288 raft_consensus.cc:1261] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [term 2 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 02:02:29.076090  3303 consensus_queue.cc:1035] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 02:02:29.078868  1018 consensus_queue.cc:237] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: false } }
I20240502 02:02:29.081163  1824 raft_consensus.cc:1261] T b1005c925b954ca39a9ee6f2677b3a1f P eb5a2343514740caa85ff0228a535123 [term 2 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 02:02:29.081460  1155 raft_consensus.cc:1261] T b1005c925b954ca39a9ee6f2677b3a1f P 8b9e878e72f64482bdd89f28ac427494 [term 2 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 02:02:29.082020  3267 consensus_queue.cc:1035] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 02:02:29.082746  3303 consensus_queue.cc:1035] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 02:02:29.084491  3267 raft_consensus.cc:2932] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER d37e50431f30463c872c5d819ddf2c25 (127.25.109.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: false } } }
I20240502 02:02:29.085239  1824 raft_consensus.cc:2932] T b1005c925b954ca39a9ee6f2677b3a1f P eb5a2343514740caa85ff0228a535123 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER d37e50431f30463c872c5d819ddf2c25 (127.25.109.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: false } } }
I20240502 02:02:29.085978  1155 raft_consensus.cc:2932] T b1005c925b954ca39a9ee6f2677b3a1f P 8b9e878e72f64482bdd89f28ac427494 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER d37e50431f30463c872c5d819ddf2c25 (127.25.109.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: false } } }
I20240502 02:02:29.089869   860 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b1005c925b954ca39a9ee6f2677b3a1f with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 02:02:29.090888   868 catalog_manager.cc:5484] T b1005c925b954ca39a9ee6f2677b3a1f P d0e4135c07ac4f2389ffe4e7379df71f reported cstate change: config changed from index 9 to 11, VOTER d37e50431f30463c872c5d819ddf2c25 (127.25.109.131) evicted. New cstate: current_term: 2 leader_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 02:02:29.095553  1269 tablet_service.cc:1514] Processing DeleteTablet for tablet b1005c925b954ca39a9ee6f2677b3a1f with delete_type TABLET_DATA_TOMBSTONED (TS d37e50431f30463c872c5d819ddf2c25 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:55198
I20240502 02:02:29.096827  3310 tablet_replica.cc:330] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25: stopping tablet replica
I20240502 02:02:29.097098  3310 raft_consensus.cc:2229] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 02:02:29.097396  3310 raft_consensus.cc:2258] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 02:02:29.098744  3310 ts_tablet_manager.cc:1875] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 02:02:29.104665  3310 ts_tablet_manager.cc:1888] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240502 02:02:29.104799  3310 log.cc:1196] T b1005c925b954ca39a9ee6f2677b3a1f P d37e50431f30463c872c5d819ddf2c25: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/wals/b1005c925b954ca39a9ee6f2677b3a1f
I20240502 02:02:29.105561   859 catalog_manager.cc:4830] TS d37e50431f30463c872c5d819ddf2c25 (127.25.109.131:37799): tablet b1005c925b954ca39a9ee6f2677b3a1f (table rebalance_test_table_1 [id=8991230805234f99bb3820fc5a537d77]) successfully deleted
I20240502 02:02:29.111639  1288 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "884b0248d66f4575a7e8e3ead42c58c6"
dest_uuid: "d37e50431f30463c872c5d819ddf2c25"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:38308
I20240502 02:02:29.111929  1288 raft_consensus.cc:595] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [term 1 LEADER]: Received request to transfer leadership
I20240502 02:02:29.243294  2567 raft_consensus.cc:979] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25: : Instructing follower d0e4135c07ac4f2389ffe4e7379df71f to start an election
I20240502 02:02:29.243486  3280 raft_consensus.cc:1067] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [term 1 LEADER]: Signalling peer d0e4135c07ac4f2389ffe4e7379df71f to start an election
I20240502 02:02:29.244220  1018 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "884b0248d66f4575a7e8e3ead42c58c6"
dest_uuid: "d0e4135c07ac4f2389ffe4e7379df71f"
 from {username='slave'} at 127.25.109.131:48671
I20240502 02:02:29.244459  1018 raft_consensus.cc:482] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 02:02:29.244565  1018 raft_consensus.cc:3037] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f [term 1 FOLLOWER]: Advancing to term 2
I20240502 02:02:29.246512  1018 raft_consensus.cc:504] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: false } }
I20240502 02:02:29.247350  1018 leader_election.cc:288] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f [CANDIDATE]: Term 2 election: Requested vote from peers 8b9e878e72f64482bdd89f28ac427494 (127.25.109.130:43631), d37e50431f30463c872c5d819ddf2c25 (127.25.109.131:37799), eb5a2343514740caa85ff0228a535123 (127.25.109.132:44441)
I20240502 02:02:29.247756  1155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "884b0248d66f4575a7e8e3ead42c58c6" candidate_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "8b9e878e72f64482bdd89f28ac427494"
I20240502 02:02:29.247910  1288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "884b0248d66f4575a7e8e3ead42c58c6" candidate_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "d37e50431f30463c872c5d819ddf2c25"
I20240502 02:02:29.248091  1155 raft_consensus.cc:3037] T 884b0248d66f4575a7e8e3ead42c58c6 P 8b9e878e72f64482bdd89f28ac427494 [term 1 FOLLOWER]: Advancing to term 2
I20240502 02:02:29.248139  1288 raft_consensus.cc:3032] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 02:02:29.248239  1288 raft_consensus.cc:729] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d37e50431f30463c872c5d819ddf2c25, State: Running, Role: LEADER
I20240502 02:02:29.248215  1824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "884b0248d66f4575a7e8e3ead42c58c6" candidate_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "eb5a2343514740caa85ff0228a535123"
I20240502 02:02:29.248543  1824 raft_consensus.cc:3037] T 884b0248d66f4575a7e8e3ead42c58c6 P eb5a2343514740caa85ff0228a535123 [term 1 FOLLOWER]: Advancing to term 2
I20240502 02:02:29.248507  1288 consensus_queue.cc:260] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: false } }
I20240502 02:02:29.248858  1288 raft_consensus.cc:3037] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [term 1 FOLLOWER]: Advancing to term 2
I20240502 02:02:29.250941  1155 raft_consensus.cc:2454] T 884b0248d66f4575a7e8e3ead42c58c6 P 8b9e878e72f64482bdd89f28ac427494 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0e4135c07ac4f2389ffe4e7379df71f in term 2.
I20240502 02:02:29.251282  1288 raft_consensus.cc:2454] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0e4135c07ac4f2389ffe4e7379df71f in term 2.
I20240502 02:02:29.251668  1824 raft_consensus.cc:2454] T 884b0248d66f4575a7e8e3ead42c58c6 P eb5a2343514740caa85ff0228a535123 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0e4135c07ac4f2389ffe4e7379df71f in term 2.
I20240502 02:02:29.251780   951 leader_election.cc:302] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 8b9e878e72f64482bdd89f28ac427494, d0e4135c07ac4f2389ffe4e7379df71f, d37e50431f30463c872c5d819ddf2c25; no voters: 
I20240502 02:02:29.252085  3267 raft_consensus.cc:2781] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 02:02:29.252238  3267 raft_consensus.cc:686] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f [term 2 LEADER]: Becoming Leader. State: Replica: d0e4135c07ac4f2389ffe4e7379df71f, State: Running, Role: LEADER
I20240502 02:02:29.252528  3267 consensus_queue.cc:237] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: false } }
I20240502 02:02:29.255498   867 catalog_manager.cc:5484] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f reported cstate change: term changed from 1 to 2, leader changed from d37e50431f30463c872c5d819ddf2c25 (127.25.109.131) to d0e4135c07ac4f2389ffe4e7379df71f (127.25.109.129). New cstate: current_term: 2 leader_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 02:02:29.656193  1288 raft_consensus.cc:1261] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [term 2 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 02:02:29.656817  3267 consensus_queue.cc:1035] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37e50431f30463c872c5d819ddf2c25" member_type: VOTER last_known_addr { host: "127.25.109.131" port: 37799 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 02:02:29.696208  1155 raft_consensus.cc:1261] T 884b0248d66f4575a7e8e3ead42c58c6 P 8b9e878e72f64482bdd89f28ac427494 [term 2 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 02:02:29.696934  3303 consensus_queue.cc:1035] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 02:02:29.701700  1824 raft_consensus.cc:1261] T 884b0248d66f4575a7e8e3ead42c58c6 P eb5a2343514740caa85ff0228a535123 [term 2 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 02:02:29.703048  3267 consensus_queue.cc:1035] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 02:02:30.278086  1018 consensus_queue.cc:237] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: false } }
I20240502 02:02:30.279976  1156 raft_consensus.cc:1261] T 884b0248d66f4575a7e8e3ead42c58c6 P 8b9e878e72f64482bdd89f28ac427494 [term 2 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 02:02:30.279958  1824 raft_consensus.cc:1261] T 884b0248d66f4575a7e8e3ead42c58c6 P eb5a2343514740caa85ff0228a535123 [term 2 FOLLOWER]: Refusing update from remote peer d0e4135c07ac4f2389ffe4e7379df71f: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 02:02:30.280580  3333 consensus_queue.cc:1035] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 02:02:30.280911  3356 consensus_queue.cc:1035] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 02:02:30.283452  3303 raft_consensus.cc:2932] T 884b0248d66f4575a7e8e3ead42c58c6 P d0e4135c07ac4f2389ffe4e7379df71f [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER d37e50431f30463c872c5d819ddf2c25 (127.25.109.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: false } } }
I20240502 02:02:30.284209  1824 raft_consensus.cc:2932] T 884b0248d66f4575a7e8e3ead42c58c6 P eb5a2343514740caa85ff0228a535123 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER d37e50431f30463c872c5d819ddf2c25 (127.25.109.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: false } } }
I20240502 02:02:30.284323  1156 raft_consensus.cc:2932] T 884b0248d66f4575a7e8e3ead42c58c6 P 8b9e878e72f64482bdd89f28ac427494 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER d37e50431f30463c872c5d819ddf2c25 (127.25.109.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: false } } }
I20240502 02:02:30.287503   860 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 884b0248d66f4575a7e8e3ead42c58c6 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 02:02:30.288728   867 catalog_manager.cc:5484] T 884b0248d66f4575a7e8e3ead42c58c6 P eb5a2343514740caa85ff0228a535123 reported cstate change: config changed from index 9 to 11, VOTER d37e50431f30463c872c5d819ddf2c25 (127.25.109.131) evicted. New cstate: current_term: 2 leader_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8b9e878e72f64482bdd89f28ac427494" member_type: VOTER last_known_addr { host: "127.25.109.130" port: 43631 } } peers { permanent_uuid: "eb5a2343514740caa85ff0228a535123" member_type: VOTER last_known_addr { host: "127.25.109.132" port: 44441 } attrs { promote: false } } peers { permanent_uuid: "d0e4135c07ac4f2389ffe4e7379df71f" member_type: VOTER last_known_addr { host: "127.25.109.129" port: 33963 } attrs { promote: false } } }
I20240502 02:02:30.296234  1269 tablet_service.cc:1514] Processing DeleteTablet for tablet 884b0248d66f4575a7e8e3ead42c58c6 with delete_type TABLET_DATA_TOMBSTONED (TS d37e50431f30463c872c5d819ddf2c25 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:55198
I20240502 02:02:30.296907  3362 tablet_replica.cc:330] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25: stopping tablet replica
I20240502 02:02:30.297098  3362 raft_consensus.cc:2229] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 02:02:30.297336  3362 raft_consensus.cc:2258] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 02:02:30.298101  3362 ts_tablet_manager.cc:1875] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 02:02:30.302582  3362 ts_tablet_manager.cc:1888] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240502 02:02:30.302711  3362 log.cc:1196] T 884b0248d66f4575a7e8e3ead42c58c6 P d37e50431f30463c872c5d819ddf2c25: Deleting WAL directory at /tmp/dist-test-taskb4pM1q/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714614746325827-26038-0/raft_consensus-itest-cluster/ts-2/wal/wals/884b0248d66f4575a7e8e3ead42c58c6
I20240502 02:02:30.303453   859 catalog_manager.cc:4830] TS d37e50431f30463c872c5d819ddf2c25 (127.25.109.131:37799): tablet 884b0248d66f4575a7e8e3ead42c58c6 (table rebalance_test_table_2 [id=57c9d0a8579e4e908683ee50f24d5aa1]) successfully deleted
I20240502 02:02:30.684996   998 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:02:30.689947  1269 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:02:30.692641  1135 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:30.702478  1804 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:02:30.849090   998 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:02:30.851068  1135 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:30.861356  1804 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:02:30.862834  1269 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:02:31.013181  1135 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:31.018303   998 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:02:31.020124  1269 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:02:31.027489  1804 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:02:31.179685  1135 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:31.189486  1269 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:02:31.192054  1804 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:02:31.194272   998 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:02:31.342757  1135 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 02:02:31.346287  1269 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:02:31.348987  1804 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 02:02:31.361094   998 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 02:02:31.569785 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 936
I20240502 02:02:31.581638  1060 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 02:02:31.754844 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 936
W20240502 02:02:31.781733  1757 connection.cc:535] server connection from 127.25.109.129:50735 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 02:02:31.781774  1089 connection.cc:535] server connection from 127.25.109.129:51057 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 02:02:31.782789 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 1070
I20240502 02:02:31.789045  1196 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 02:02:31.957594 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 1070
I20240502 02:02:31.982945 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 1204
I20240502 02:02:31.988880  1328 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 02:02:32.147058 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 1204
I20240502 02:02:32.172415 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 1742
I20240502 02:02:32.178457  1865 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 02:02:32.325646 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 1742
W20240502 02:02:32.348050   860 connection.cc:535] server connection from 127.25.109.132:39889 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 02:02:32.348182 26038 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu pid 844
I20240502 02:02:32.349268   906 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 02:02:32.515431 26038 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskb4pM1q/build/asan/bin/kudu with pid 844
2024-05-02T02:02:32Z chronyd exiting
[       OK ] LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer/idx1_rf3_t3_l_3_1 (27736 ms)
[----------] 2 tests from LocationAwareRebalancingParamTest (46071 ms total)

[----------] Global test environment tear-down
[==========] 7 tests from 6 test suites ran. (606219 ms total)
[  PASSED  ] 6 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/0, where GetParam() = 4-byte object <00-00 00-00>

 1 FAILED TEST